Set Variables for Airflow during Helm install-CodePudding

I'm installing Airflow on kind with the following command:

export RELEASE_NAME=first-release
export NAMESPACE=airflow

helm install $RELEASE_NAME apache-airflow/airflow --namespace $NAMESPACE \
    --set images.airflow.repository=my-dags \
    --set images.airflow.tag=0.0.1 \
    --values env.yaml

Andthe file env.yaml looks like the below:

env:
  - name: "AIRFLOW_VAR_KEY"
    value: "value_1"

But from the Web UI (when I go to Admins --> Variables), these credentials don't appear there.

How do I pass these credentials during helm install? Thanks!

UPDATE: It turns out that the environment variable was set successfully. However it doesn't show up on the Web UI

CodePudding user response：

i am not sure how your full env.yaml file is

but to set the environment variables in Airflow

## environment variables for the web/scheduler/worker Pods (for airflow configs)
  ##
  ## WARNING:
  ## - don't include sensitive variables in here, instead make use of `airflow.extraEnv` with Secrets
  ## - don't specify `AIRFLOW__CORE__SQL_ALCHEMY_CONN`, `AIRFLOW__CELERY__RESULT_BACKEND`,
  ##   or `AIRFLOW__CELERY__BROKER_URL`, they are dynamically created from chart values
  ##
  ## NOTE:
  ## - airflow allows environment configs to be set as environment variables
  ## - they take the form: AIRFLOW__<section>__<key>
  ## - see the Airflow documentation: https://airflow.apache.org/docs/stable/howto/set-config.html
  ##
  ## EXAMPLE:
  ##   config:
  ##     ## Security
  ##     AIRFLOW__CORE__SECURE_MODE: "True"
  ##     AIRFLOW__API__AUTH_BACKEND: "airflow.api.auth.backend.deny_all"

Reference file

and after that you have to run your command and further your DAG will be able to access the variables.

Helm documentation : https://github.com/helm/charts/tree/master/stable/airflow#docs-airflow---configs

Make sure you are configuring section : airflow.config

CodePudding user response：

Okay, so I've figured this one out: The environment variables are set just nicely on the pods. However, this will not appear on the Web UI.

Workaround: to make it appear in the web UI, I will have to go into the Scheduler pod and import the variables. It can be done with a bash script.

# Get the name of scheduler pod
export SCHEDULER_POD_NAME="$(kubectl get pods --no-headers -o custom-columns=":metadata.name" -n airflow | grep scheduler)"
# Copy variables to the scheduler pod
kubectl cp ./variables.json airflow/$SCHEDULER_POD_NAME:./
# Import variables to scheduler with airflow command
kubectl -n $NAMESPACE exec $SCHEDULER_POD_NAME -- airflow variables import variables.json