How to add airflow variables in docker compose file?-CodePudding

I have an docker compose file which spins up the local airflow instance as below:

version: '3.7'
services:
    postgres:
        image: postgres:9.6
        environment:
            - POSTGRES_USER=airflow
            - POSTGRES_PASSWORD=airflow
            - POSTGRES_DB=airflow
        logging:
            options:
                max-size: 10m
                max-file: "3"

    webserver:
        image: puckel/docker-airflow:1.10.6
        restart: always
        depends_on:
            - postgres
        environment:
            - LOAD_EX=n
            - EXECUTOR=Local
            - FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
        logging:
            options:
                max-size: 10m
                max-file: "3"
        volumes:
            - ./dags:/usr/local/airflow/dags
            - ${HOME}/.aws:/usr/local/airflow/.aws
        ports:
            - "8080:8080"
        command: webserver
        healthcheck:
            test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
            interval: 30s
            timeout: 30s
            retries: 3

I want to add some Airflow variables which the underlying dag uses eg: CONFIG_BUCKET. I have added them as AIRFLOW_VAR_CONFIG_BUCKET=s3://foo-bucket in environment section of web server but it does not seem to work. Any ideas how can I achieve this ?

CodePudding user response：

If you add an environment variable named AIRFLOW_VAR_CONFIG_BUCKET to the list under environment:, it should be accessible by Airflow. Sounds like you're doing that correctly.

Two things to note:

Variables (& connections) set via environment variables are not visible in the Airflow UI. You can test if they exist by executing Variable.get("config_bucket") in code.
The Airflow scheduler/worker (depending on Airflow executor) require access to the variable while running a task. Adding a variable to the webserver is not required.

CodePudding user response：

You should not add variables to the webserver, but to scheduler. If you are using LocalExecutor, the tasks are run in the context of Scheduler.

Actually what tou should really do is to set all env variables to be the same for all the containers (this is explained here https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html)

Use the same configuration across all the Airflow components. While each component does not require all, some configurations need to be same otherwise they would not work as expected. A good example for that is secret_key which should be same on the Webserver and Worker to allow Webserver to fetch logs from Worker.

There are a number of ways you can do it - just read the docker-compose documntation on that https://docs.docker.com/compose/environment-variables . You can also see the "Quick start" docker compose from Airflow docs where we used anchors - which is bit more sphisticated way https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html

Just note that the "quick start" should be just inspiration, it is nowhere near production setup and if you want to make your own docker compose you need to really get a deeper understanding of the docker compose - as warned in the note in our docs.