Following the MySql guide here, I wrote Here the docker-compose file. First I spin-up airflow-init container that runs db migration and creates admin user in mysql database. Then run scheduler, I start seeing following logs in MySql;

Aborted connection 11 to db: 'airflow' user: 'airflow' host: '192.168.0.3' (Got an error reading communication packets)

More of these logs, after running webserver too. However, I am able to login to UI and trigger DAGs as usual, but the UI is very slow (as compared to when using SqLite db with sequential executor). and i mostly get The scheduler does not appear to be running. Last heartbeat was received <x seconds/minutes> ago. as a warning in UI. I tried increasing max_allowed_packet, max_connections as well as interactive_timeout as mentioned in here, but no luck. I've also doubled the sql_alchemy_pool_recycle to 3600.

This issue was reported in earlier versions of airflow here. I am expecting the UI to be running smooth. Has anyone faced the issue ? Any hints or clues ?

CodePudding user response：

There is not too much information, but Airflow with MySQL should be WAY faster than Sqlite.

What happened most likely is that you have not enough memory allocated to the docker engine and MySQL/Airflow started to use swap disk.

I believe by default MySQL container uses at least 800 MB if you don't configure it to use less and together with running Airflow they can only be comfortable with ~ 4GB.

If you are on MacOS (guessing), default memory for Docker Engine is 2GB I believe, so you need to increase it.