Getting ModuleNotFoundError even though I already have __init_

I am working on an Airflow-Python project. I have created a utils folder, and inside that I am creating a utility class to be used by different DAG's. I am not getting any error in my IDE(Pycharm), but on the Airflow DAG's UI getting ModuleNotFoundError: No module named 'utils'

Here is my project structure -

Inside my utils/init.py -

from .kairos_bq_to_mysql import BigQueryToMySql

Inside utils/kairos_bq_to_mysql.py -

It contains the actual class definition.

I am trying to import this class inside dags/bq_to_mysql/umc_cg_service_sync.py I am trying to import BigQueryToMySql class.

from utils import BigQueryToMySql

I don't get any error/warning in PyCharm. The class navigation also works fine. But on the Airflow UI Dashboard, I am getting error.

Can someone please help me fix this issue.

In my docker-compose.yml

  volumes:
    - ./dags:/opt/airflow/dags
    - ./logs:/opt/airflow/logs
    - ./plugins:/opt/airflow/plugins
    - ./configs:/opt/airflow/configs
    - ./utils:/opt/airflow/utils

Do I need make any changes in my docker compose file ?

CodePudding user response：

You would need to import utils from the directory above the two packages, utils and dags.

from kairos-aggs-airflow.utils import BigQueryToMySql

However, that directory, kairos-aggs-airflow has hyphens and won't work. So either rename that directory or put the packages in a new directory.

CodePudding user response：

Use sys.path.insert(0, os.path.dirname(__file__)) this before your local import like

sys.path.insert(0, os.path.dirname(__file__))
from utils import BigQueryToMySql

This addes your dag python file into system variable worked for me