I have three tasks those need to be executed for 100s of users. Eaxample users lsit:
['user1', 'user2', 'user3','user4', 'user5', 'user6']
Here is the code snippet for celery beat:
for user in users:
APP.conf.beat_schedule = {
"trigger-sync_books": {
"task": "call_sync_books",
"schedule": crontab(minute=0, hour="*/24"),
'args': (user, bo, gg,),
},
"trigger-sync_stock": {
"task": "call_sync_stock",
"schedule": crontab(minute="*/10"),
'args': (user, bo, gg,),
},
"trigger-sync_customer": {
"task": "call_sync_customer",
"schedule": crontab(minute=0, hour="*/24"),
'args': (user, bo, gg,),
},
}
But everytime the celery beat sends these three tasks for the last user from the list.
i.e., for user6
from the example list.
How can I make sure these three tasks will be excuted for all the users?
CodePudding user response:
This will not work. You are redefining APP.conf.beat_schedule as many times as you have users, so basically all previous 5 configs get overwritten by the last one in that loop.
Try the following:
for user in users:
us_sched = {
f"trigger-sync_books-{user}": {
"task": "call_sync_books",
"schedule": crontab(minute=0, hour="*/24"),
'args': (user, bo, gg,),
},
f"trigger-sync_stock-{user}": {
"task": "call_sync_stock",
"schedule": crontab(minute="*/10"),
'args': (user, bo, gg,),
},
f"trigger-sync_customer-{user}": {
"task": "call_sync_customer",
"schedule": crontab(minute=0, hour="*/24"),
'args': (user, bo, gg,),
}
}
# Python >= 3.5:
APP.conf.beat_schedule = {**APP.conf.beat_schedule, **us_sched}
# Python 3.9: APP.conf.beat_schedule |= us_sched
By the time the loop above ends you wil have all schedules for all users in the APP.conf.beat_schedule dictionary.