Changing default Airflow configuration options (run on Docker)-CodePudding

I've been trying to set up an Airflow instance on a set of docker containers for the sake of learning. I know just the core basics of both Docker and Airflow.

I've been following this guide on how to set up Airflow using Docker.

When following the guide I can start up the containers and everything work as expected.

My next step was to start changing different configuration alternatives in the docker-compose.yml-file. I started by trying to set different usernames and passwords.

I created an .env-file in the same directory as the docker-compose.yml-file. See below.

AIRFLOW_UID=50000
USER='user'
PASSWORD='password'
DATABASE='airflow'
_AIRFLOW_WWW_USER_USERNAME='user'
_AIRFLOW_WWW_USER_PASSWORD='password'

In order to use these environments variables, I made some changes to the connection strings in the docker-compose.yml aswell. Original lines has been commented out. Se below.

x-airflow-common:
  &airflow-common
  # In order to add custom dependencies or upgrade provider packages you can use your extended image.
  # Comment the image line, place your Dockerfile in the directory where you placed the docker-compose.yaml
  # and uncomment the "build" line below, Then run `docker-compose build` to build the images.
  image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.4.2}
  # build: .
  environment:
    &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
    AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql psycopg2://${USER}:${PASSWORD}@postgres/${DATABASE}
    # For backward compatibility, with Airflow <2.3
    AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql psycopg2://${USER}:${PASSWORD}@postgres/${DATABASE}
    AIRFLOW__CELERY__RESULT_BACKEND: db postgresql://${USER}:${PASSWORD}@postgres/${DATABASE}
# -- Original settings --
#AIRFLOW__DATABASE__SQL_ALCHEMY_CONN:postgresql psycopg2://airflow:airflow@postgres/airflow
#AIRFLOW__CORE__SQL_ALCHEMY_CONN:postgresql psycopg2://airflow:airflow@postgres/airflow
#AIRFLOW__CELERY__RESULT_BACKEND: db postgresql://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
    AIRFLOW__CORE__FERNET_KEY: ''
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'true'
    AIRFLOW__API__AUTH_BACKENDS: 'airflow.api.auth.backend.basic_auth'
    _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
  volumes:
    - ./dags:/opt/airflow/dags
    - ./logs:/opt/airflow/logs
    - ./plugins:/opt/airflow/plugins
  user: "${AIRFLOW_UID:-50000}:0"
  depends_on:
    &airflow-common-depends-on
    redis:
      condition: service_healthy
    postgres:
      condition: service_healthy

services:
  postgres:
    image: postgres:13
    environment:
      POSTGRES_USER: ${USER}
      POSTGRES_PASSWORD: ${PASSWORD}
      POSTGRES_DB: ${DATABASE}
     # -- Original settings --
     #POSTGRES_USER: airflow
     #POSTGRES_PASSWORD: airflow
     #POSTGRES_DB: airflow
    volumes:
      - postgres-db-volume:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD", "pg_isready", "-U", '${USER}']
      interval: 5s
      retries: 5
    restart: always

First I run docker-compose down --volumes --rmi all in order to clean up the environment. I then run docker-compose up airflow-init. Below are the last lines displayed in the terminal.

data_collection_proj-airflow-init-1  | User "user" created with role "Admin"
data_collection_proj-airflow-init-1  | /home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py:367: FutureWarning: The auth_backends setting in [api] has had airflow.api.auth.backend.session added in the running config, which is needed by the UI. Please update your config before Apache Airflow 3.0.
data_collection_proj-airflow-init-1  |   FutureWarning,
data_collection_proj-airflow-init-1  | 2.4.2
data_collection_proj-airflow-init-1 exited with code 0

No errors or warnings were displayed so it seems to work fine.

I then run docker-compose up and get the following error.

data_collection_proj-postgres-1           | 2022-11-12 17:47:08.939 UTC [165] FATAL:  database "user" does not exist

My changes seem to destroy the set-up but I don't understand why.

CodePudding user response：

You can try this test syntax, it works fine for me:

services:
  postgres:
    image: postgres:13
    environment:
      POSTGRES_USER: ${USER}
      POSTGRES_PASSWORD: ${PASSWORD}
      POSTGRES_DB: ${DATABASE}
    volumes:
      - postgres-db-volume:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "sh -c 'pg_isready -U ${USER} -d ${DATABASE}'"]
      interval: 5s
      retries: 5
    restart: always