Problem: new dags not shown on docker airflow, no error when running airflow dags list-import-errors

Docker image:

However, the dags is still not shown on both webserver UI and airflow dags list. Running airflow dags list-import-errors also yield no result.

When I open the docker terminal, I can see my dag inside the dags folder via ls command. I also tried to make the owner root by using chown, but both of my dag still not shown up on the list.

The airflow run successfully (via docker compose) as I can see example dags, but not my own dags.

Any help will be appreciated. Thanks!

CodePudding user response：

I would try a few things:

• rename files to sth like gcp_dag.py and python_dag.py
• ensure import airflow is present in each file
• ensure you create DAG object in each file
• add __init__.py (empty) file to dags folder

It would be also helpful to see contents of at least one of those files.

CodePudding user response：

1. do not name your python DAGs test_* that probably does not matter but this is usually a convention for unit tests
2. make sure there is a DAG word in your dags (or change https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dag-discovery-safe-mode to False) - the DAGs can be ignored if they do not contain "DAG" or "airflow"
3. Exec into your scheduler container and check if it can see and read the DAGs and whether the env vars are properly set airflow info run in your scheduler container (providing that it is run with the same env as the running scheduler) should show all the configuration
4. Check if your scheduler is runnning
5. look at the logs of the scheduler and see if it is scanning the right folders - enabling DEBUG log might be helpful https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#logging-level
6. see if you can see the dags with airflow dags sub commands