I am running a celery worker and celery beat with DatabaseScheduler.
I am running these on Heroku along with a django web application. They are being used to send emails and texts at regular intervals, Once per day at midnight for emails and 730am for texts.
I have defined the tasks manually using a chrontab in Periodic Tasks table, but I have also tried defining them in the codebase using celeryapp.conf.beat_schedule, both behave the same.
The tasks look like this
periodic tasks in django admin
The issue I am having is that Heroku restarts its dynos once per day as a policy, and when this happens, the celery beat dyno runs my periodic tasks for some reason. The debug log reads "Scheduler: Sending due task" as if it is normal, whether they are due then or not. I do not know why this is happening.
I have celery results running as well with postgres backed task results and that looks like this, where I have the tasks running at the correct times (midnight and 730am), but also at some random time when heroku restarts the dynos.
How can I stop the tasks from running when this restart happens? Why are they running at all after the restart?
EDIT: I was able to manually restart using heroku ps:restart beat and watch it happen, still don't know why but here is the debug log where it occurs. Keep in mind the tasks have already run successfully for this day and the last_run was correct prior to restarting. Also, restarting again DOES NOT run the tasks again. I wonder if somehow sometimes the restart doesn't see the last_run date? Still trying to figure this out.
2022-10-15T16:22:33.824655 00:00 app[beat.1]: [2022-10-15 12:22:33,824: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
2022-10-15T16:22:38.828379 00:00 app[beat.1]: [2022-10-15 12:22:38,828: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
2022-10-15T16:22:43.832789 00:00 app[beat.1]: [2022-10-15 12:22:43,832: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
2022-10-15T16:22:46.627485 00:00 heroku[beat.1]: Restarting
2022-10-15T16:22:46.630753 00:00 heroku[beat.1]: State changed from up to starting
2022-10-15T16:22:48.836424 00:00 app[beat.1]: [2022-10-15 12:22:48,836: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
2022-10-15T16:22:48.894489 00:00 heroku[beat.1]: Stopping all processes with SIGTERM
2022-10-15T16:22:48.956144 00:00 app[beat.1]: [2022-10-15 12:22:48,956: DEBUG/MainProcess] Writing entries...
2022-10-15T16:22:48.956272 00:00 app[beat.1]: [2022-10-15 12:22:48,956: DEBUG/MainProcess] Writing entries...
2022-10-15T16:22:48.956533 00:00 app[beat.1]: [2022-10-15 12:22:48,956: DEBUG/MainProcess] Writing entries...
2022-10-15T16:22:48.956585 00:00 app[beat.1]: [2022-10-15 12:22:48,956: DEBUG/MainProcess] Writing entries...
2022-10-15T16:22:49.225748 00:00 heroku[beat.1]: Process exited with status 0
2022-10-15T16:22:50.122842 00:00 heroku[beat.1]: Starting process with command `celery -A hello beat -l debug --scheduler django_celery_beat.schedulers:DatabaseScheduler`
2022-10-15T16:22:50.807227 00:00 heroku[beat.1]: State changed from starting to up
2022-10-15T16:22:51.834963 00:00 app[beat.1]: celery beat v5.2.7 (dawn-chorus) is starting.
2022-10-15T16:22:52.276824 00:00 app[beat.1]: __ - ... __ - _
2022-10-15T16:22:52.276844 00:00 app[beat.1]: LocalTime -> 2022-10-15 12:22:52
2022-10-15T16:22:52.276845 00:00 app[beat.1]: Configuration ->
2022-10-15T16:22:52.276845 00:00 app[beat.1]: . broker -> redis://:**@ec2-54-165-246-156.compute-1.amazonaws.com:31259//
2022-10-15T16:22:52.276846 00:00 app[beat.1]: . loader -> celery.loaders.app.AppLoader
2022-10-15T16:22:52.276846 00:00 app[beat.1]: . scheduler -> django_celery_beat.schedulers.DatabaseScheduler
2022-10-15T16:22:52.276846 00:00 app[beat.1]:
2022-10-15T16:22:52.276846 00:00 app[beat.1]: . logfile -> [stderr]@