I've been runnning into what should be a simple issue with my airflow scheduler. Every couple of weeks, the scheduler becomes Evicted. When I run a describe on the pod, the issue is because The node was low on resource: ephemeral-storage. Container scheduler was using 14386916Ki, which exceeds its request of 0.

The question is two fold. First, why is the scheduler utilizing ephemeral-storage? And second, is it possible to do add ephemeral-storage when running on eks?

Thanks!

CodePudding user response：

I believe Ephemeral Storage is not Airflow's question but more of the configuration of your K8S cluster.

Assuming we are talking about OpenShift' ephemeral storage:

https://docs.openshift.com/container-platform/4.9/storage/understanding-ephemeral-storage.html

This can be configured in your cluster and it wil make "/var/log" ephemeral.

I think the problem is that /var/logs gets full. Possibly some of the system logs (not from airlfow but from some other processes running in the same container). I think a solution will be to have a job that cleans that system log periodically.

For example we have this script that cleans-up Airlfow logs:

https://github.com/apache/airflow/blob/main/scripts/in_container/prod/clean-logs.sh