Airflow Kubernetes Executor not creating pods-CodePudding

I'm trying to create a setup for Airflow with Kubernetes Executor. However I am having an issue where Airflow is not able to create pods when executing tasks. However the initial setup is able to create the constituent parts just fine.

Please see https://github.com/neku-dev-88/airflow_kubernetes for the code on airflow value files and kubectl resources.

The way I set this up locally installs the following steps in command prompt (Windows, I use kind: https://kind.sigs.k8s.io/):

kind create cluster --image kindest/node:v1.18.15
helm repo add airflow-stable https://airflow-helm.github.io/charts
kubectl apply -n airflow-staging -f roles.yaml
helm install airflow-staging airflow-stable/airflow -f charts/airflow/values-staging.yaml --namespace airflow-staging

When I try running a DAG it just stays in "running" status: in running status

I see no new pods are being created: no new pods

I have even created an airflow service role and used a cluster role binding to give it a cluster role to create pods, but no luck.

values file:

###################################
## CONFIG | Kubernetes ServiceAccount
###################################
serviceAccount:
  ## if a Kubernetes ServiceAccount is created
  ## - if `false`, you must create the service account outside this chart with name: `serviceAccount.name`
  ##
  create: true

  ## the name of the ServiceAccount
  ## - by default the name is generated using the `airflow.serviceAccountName` template in `_helpers/common.tpl`
  ##
  name: airflow

roles file:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: pods-permissions
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch", "create", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: airflow-pods
  namespace: airflow-staging
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: pods-permissions
subjects:
  - kind: ServiceAccount
    name: airflow
    namespace: airflow-staging
---

I feel like I'm missing something in the setup, please advise.

CodePudding user response：

I figured it out, the problem was that my DAGs were all paused and needed to be unpaused.