I'm writing an Airflow DAG using the KubernetesPodOperator
. A Python process running in the container must open a file with sensitive data:
with open('credentials/jira_credentials.json', 'r') as f:
creds = json.load(f)
and a CloudStorage client must be authenticated:
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = "credentials/cloud_storage_credentials.json"
According to the best security practices I don't package a container's image with sensitive data. Instead I use Kubernetes Secrets. Using Python API for Kubernetes I'm trying to mount them as a volume but with no success. The credentials/
directory exists in the container but it's empty. What should I do to make files jira_crendentials.json and cloud_storage_credentials.json accessible in the container?
My DAG's code:
from airflow import DAG
from datetime import datetime, timedelta
from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
from airflow.kubernetes.secret import Secret
from airflow.kubernetes.volume import Volume
from airflow.kubernetes.volume_mount import VolumeMount
from airflow.operators.dummy_operator import DummyOperator
from kubernetes.client import models as k8s
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime.utcnow(),
'email': ['[email protected]'],
'email_on_failure': False,
'email_on_retry': False,
'retry_delay': timedelta(minutes=5)
}
volume = Volume(name="volume-credentials", configs={})
volume_mnt = VolumeMount(mount_path="/credentials", name="volume-credentials", sub_path="", read_only=True)
secret_jira_user = Secret(deploy_type="volume",
deploy_target="/credentials",
secret="jira-user-secret",
key="jira_credentials.json")
secret_storage_credentials = Secret(deploy_type="volume",
deploy_target="/credentials",
secret="jira-trans-projects-cloud-storage-creds",
key="cloud_storage_credentials.json")
dag = DAG(
dag_id="jira_translations_project",
schedule_interval="0 1 * * MON",
start_date=datetime(2021, 9, 5, 0, 0, 0),
max_active_runs=1,
default_args=default_args
)
start = DummyOperator(task_id='START', dag=dag)
passing = KubernetesPodOperator(namespace='default',
image="eu.gcr.io/data-engineering/jira_downloader:v0.18",
cmds=["/usr/local/bin/run_process.sh"],
name="jira-translation-projects-01",
task_id="jira-translation-projects-01",
get_logs=True,
dag=dag,
volumes=[volume],
volume_mounts=[volume_mnt],
secrets=[
secret_jira_user,
secret_storage_credentials],
env_vars={'MIGRATION_DATETIME': '2021-01-02T03:04:05'},
)
start >> passing
CodePudding user response:
According to this example, Secret
is a special class that will handle creating volume mounts automatically. Looking at your code, seems that your own volume with mount /credentials
is overriding /credentials
mount created by Secret
, and because you provide empty configs={}
, that mount is empty as well.
Try supplying just secrets=[secret_jira_user,secret_storage_credentials]
and removing manual volume_mounts
.