I am playing around with containers (docker) and I'm trying to create several extraction jobs.
Long story short, currently I have a huge container that runs for several hours doing these extractions. I want to break this docker image in several minor images, but there is a catch: some of these extractions need to run sequentially. Meaning, for example, that in some cases a given container can only start after a previous one ends.
Is it possible to do this with docker compose? Is there any other solution? I would like to avoid kubernetes and run this on aws or azure ( with azure container instances or aws ecs)?
Thanks
CodePudding user response：
First of all, split your application into as many components as you need. Ideally, one container should carry out one task only.
Then, given that you want to run the containers sequentially, you should adopt a state machine service. On AWS, you can use Step Functions. On Azure, you can use Logic Apps. With either of these services, you can configure running containers in sequence or even in parallel, as needed.
CodePudding user response：
You could use Amazon Managed Workflows for Apache Airflow (MWAA) to create and run these workflows. This products is made for creating end-to-end data pipelines.
You can use container plugins to use ECS or EKS:
In addition, it is possible to define and run data operation tasks even without using containers and use DAGs with other plugin packages instead. See for example, Providers packages reference