I'm trying to learn a little Airflow by running the official airflow docker compose image [ per instructions here: https://airflow.apache.org/docs/apache-airflow/2.0.1/start/docker.html ]. I was able to launch all the services such that when I dropped my test DAG in the ./dags folder I eventually saw my DAG in the UI and could run it.
But this took several minutes. And continues to take several minutes every time I 'deploy' a new dag into ./dags.
I'm wondering if there is some configuration option that would cause the scanning of the ./dags folder to happen faster so I don't have to wait around as much. Thanks in advance !
CodePudding user response：
As far as I know there are several reasons for slowness of your DAG's loading.
- If you have a lot (thousands) of DAGs (dynamic DAGs are slower, depends on how you generate them).
- Your DAG is too complex - see Reducing DAG complexity.
- You can test your code speed - DAG Loader Test
CodePudding user response：
In airflow.cfg file you may want to change this, default is 5 minutes.
dag_dir_list_interval = 300
After changing the value, just restart webserver, scheduler, triggerer, worker docker containers i.e. without killing them or doing docker-compose down. e.g.
docker restart your_airflow_webserver_container_id