Apache Airflow configuration files: Environment variables in docker-compose file doesn't work-CodePudding

In apache airflow configuration I tried to change Sequential executor to Celery executory using Environment variables in docker-compose files:

version: '3'
x-airflow-common:
  &airflow-common
  # In order to add custom dependencies or upgrade provider packages you can use your extended image.
  # Comment the image line, place your Dockerfile in the directory where you placed the docker-compose.yaml
  # and uncomment the "build" line below, Then run `docker-compose build` to build the images.
  #image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.2.1}
  build: .
  environment:
    &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
    AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql psycopg2://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__RESULT_BACKEND: db postgresql://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
    AIRFLOW__CORE__FERNET_KEY: ''
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
    AIRFLOW__API__AUTH_BACKEND: 'airflow.api.auth.backend.basic_auth'
    _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}

But when I check config_file it's still Sequential Executor.

First Question: How can I change it to CeleryExecutor?

Second Question: In general, how should I configure Apache Airflow: through environment variables or through config file?

Third Question: if through config file, then should I mount config file or there is another better solution?

CodePudding user response：

The variables will NOT change config file. They override the values from there. Airflow when reading configuration value will first try to see if there ia an environment variable defined - if it is, then it will use it INSTEAD of the config file. The config file will remain as it is.

See https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html and you will find "order of precedence" of reading configuration by Airflow.

The way can verify if the variable is properly set is by executing airflow config list CLI command https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#list

It will show you the configuration that is "effective" - i.e. what Airlfow sees after processing all the possibilities of reading the config values following the order of precedence. This way you can verify if you have not made any mistake.

With the quick-start docker compose you can run the airflow CLI commands as described https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html#running-the-cli-commands