I'm reading Prefect documentation and trying to understand how local deployment works. I can deploy a flow locally following the below steps.
First, I build the flow:
prefect deployment build ./log_flow.py:log_flow -n log-simple -q test
Where ./log_flow.py:log_flow are, respectively, the flow's location and entrypoint. log-simple is the name of deployment and test is the work queue
Second, I start the worker using:
prefect agent start -q 'test'
To apply the deployment, I use python running the below snippet:
from log_flow import log_flow
from prefect.deployments import Deployment
deployment = Deployment.build_from_flow(
flow=log_flow,
name="log-simple",
parameters={"name": "Marvin"},
infra_overrides={"env": {"PREFECT_LOGGING_LEVEL": "DEBUG"}},
work_queue_name="test",
)
if __name__ == "__main__":
deployment.apply()
Well, that works fine for a single flow. But how can I deploy several flows at once? I can repeat the above process for every flow, but it looks a bit unpractical to me since each build step generates another YAML file. I think would be more practical if my deployment generates a single YAML file for all flows.
Is there a way to deploy several flows at once in Prefect 2.0?
CodePudding user response:
Yes, absolutely. You could loop over all flows from Python, bash or from CI/CD. Here are a couple of examples showing how you could approach it:
- CI/CD: https://github.com/anna-geller/dataflow-ops/blob/main/.github/workflows/main.yaml#L35
- bash script (there are several example bash scripts in this repo for various infra/storage options): https://github.com/anna-geller/prefect-dataplatform/blob/main/deploy_locally.py