I have a project with the following structure:

backend
|-app # fastapi and celery
|-scrapper # selenium and celery

I am using celery to run long and short tasks, so I will have multiple queues/workers. Also, I will have tasks called from app, but that will be processed by scrapper. I am thinking how to split things among containers and I am not sure how to proceed. Here is what I am thinking to do:

• one container to run fastapi
• one container for each worker related to the fastapi one. Each container here will be pretty much a copy of the fastapi one, but the entrypoint will be to run the celery worker
• one container to run the scrapper module
• one container for each worker related to the scrapper one. Each container here will be pretty much a copy of the scrapper one, but the entrypoint will be to run the celery worker

I am kinda new to docker and this seems to be a waste of resources (to have multiple copies of the same thing, one to run fastapi and another to run celery alone), but if I understood right this is the way to go. I that right?

CodePudding user response：

I would actually have four different containers:

• fastapi
• fastapi-celery (worker(s) run with -Q fastapi)
• scraper
• scraper-celery (worker(s) run with -Q scraper)

Reason for this is simply the ability to manage and scale easily any piece of your infrastructure. At some point you may find that you do not have enough workers to handle heavy fastapi load - then you would just add more fastapi-celery containers, etc.