I am new to Kubernetes, and trying to get apache airflow working using helm charts. After almost a week of struggling, I am nowhere - even to get the one provided in the apache airflow documentation working. I use Pop OS 20.04 and microk8s.

When I run these commands:

kubectl create namespace airflow
helm repo add apache-airflow https://airflow.apache.org
helm install airflow apache-airflow/airflow --namespace airflow

The helm installation times out after five minutes.

kubectl get pods -n airflow

shows this list:

NAME                                   READY   STATUS     RESTARTS   AGE
airflow-postgresql-0                   0/1     Pending    0          4m8s
airflow-redis-0                        0/1     Pending    0          4m8s
airflow-worker-0                       0/2     Pending    0          4m8s
airflow-scheduler-565d8587fd-vm8h7     0/2     Init:0/1   0          4m8s
airflow-triggerer-7f4477dcb6-nlhg8     0/1     Init:0/1   0          4m8s
airflow-webserver-684c5d94d9-qhhv2     0/1     Init:0/1   0          4m8s
airflow-run-airflow-migrations-rzm59   1/1     Running    0          4m8s
airflow-statsd-84f4f9898-sltw9         1/1     Running    0          4m8s
airflow-flower-7c87f95f46-qqqqx        0/1     Running    4          4m8s

Then when I run the below command:

kubectl describe pod airflow-postgresql-0 -n airflow

I get the below (trimmed up to the events):

Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  58s (x2 over 58s)  default-scheduler  0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.

Then I deleted the namespace using the following commands

kubectl delete ns airflow

At this point, the termination of the pods gets stuck. Then I bring up the proxy in another terminal:

kubectl proxy

Then issue the following command to force deleting the namespace and all it's pods and resources:

kubectl get ns airflow -o json | jq '.spec.finalizers=[]' | curl -X PUT http://localhost:8001/api/v1/namespaces/airflow/finalize -H "Content-Type: application/json" --data @-

Then I deleted the PVC's using the following command:

kubectl delete pvc --force --grace-period=0 --all -n airflow

You get stuck again, so I had to issue another command to force this deletion:

kubectl patch pvc data-airflow-postgresql-0 -p '{"metadata":{"finalizers":null}}' -n airflow

The PVC's gets terminated at this point and these two commands return nothing:

kubectl get pvc -n airflow
kubectl get all -n airflow

Then I restarted the machine and executed the helm install again (using first and last commands in the first section of this question), but the same result.

I executed the following command then (using the suggestions I found here):

kubectl describe pvc -n airflow

I got the following output (I am posting the event portion of PostgreSQL):

Type    Reason         Age                   From                         Message
  ----    ------         ----                  ----                         -------
  Normal  FailedBinding  2m58s (x42 over 13m)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

So my assumption is that I need to provide storage class as part of the values.yaml

Is my understanding right? How do I provide the required (and what values) in the values.yaml?

CodePudding user response：

If you installed with helm, you can uninstall with helm delete airflow -n airflow.

Here's a way to install airflow for testing purposes using default values:

Generate the manifest helm template airflow apache-airflow/airflow -n airflow > airflow.yaml

Open the "airflow.yaml" with your favorite editor, replace all "volumeClaimTemplates" with emptyDir. Example:

Create the namespace and install:

kubectl create namespace airflow
kubectl apply -f airflow.yaml --namespace airflow

You can copy files out from the pods if needed.

To delete kubectl delete -f airflow.yaml --namespace airflow.