Where do I put a custom config (.cfg) file in AWS MWAA Airflow?-CodePudding

I have a config file, dev.cfg, that looks like this:

[S3]
bucket_name = my-bucket

I need this in my code to do S3 things. I do not have the access (nor will I be given it) to modify the environmental or config variables in the AWS console. My only method of putting files in S3 is the CLI (aws s3 cp ...).

This is the project directory structure in S3:

my-bucket/
  dags/
    dev.cfg
    some_dag_file.py
  plugins.zip
  requirements.txt

In the plugins.zip file, there is a plugin that should set the path to dev.cfg to an env var (DEV_CONFIG_PATH) that my code uses:

import os

from airflow.plugins_manager import AirflowPlugin

os.environ["DEV_CONFIG_PATH"] = os.path.join(
    os.getenv('AIRFLOW_HOME'), 'dags', 'dev.cfg')


class EnvVarPlugin(AirflowPlugin):
    name = 'env_var_plugin'

However, I'm getting an import error in the Airflow UI:

configparser.NoSectionError: No section: 'S3'

Any help is appreciated.

I'm using MWAA with Airflow version 2.4.3 running python 3.10.

CodePudding user response：

Try asking your infra team to update the config: core.lazy_load_plugins : False