I am working with Amazon Managed Workflows for Apache Airflow (MWAA).
We have some DAGs running successfully in the instance. They use the bash operator and the python operator. We do not currently have a requirments.txt file specified for the instance as there are no python external module dependencies that are not included by default in MWAA. I need to deploy a new DAG which will reference a number of external python libraries that are not included by default in MWAA. For this, I will have to specify a requirments.txt file for the MWAA instance.
What is the risk of setting a requirements.txt file on a running instance? If there is a problem with the file such as:
- bad file format
- unknown library - not available on pypi.org
- library version not available on pypi.org
- library conflict with some other library
would the existing DAGs that were already working prior to the introduction of the requirements.txt file keep working, or could this file break the running system?
I am trying to understand if it is safe to test out / deploy a requirments.txt to a live production environment. Unfortunately, I don't have a non-production environment available nor access/permission to create one in the relevant AWS account.
CodePudding user response：
What is the risk of setting a requirements.txt file on a running instance?
In general, yes, all of the problems you listed (e.g. bad file format, unknown library, etc) may cause problems with existing DAGs. This includes errors such as incorrect package versions, which may install correctly, but may not be implemented correctly.
Example error: https://docs.aws.amazon.com/mwaa/latest/userguide/t-apache-airflow-11012.html#troubleshooting-dependencies
I am trying to understand if it is safe to test out / deploy a requirments.txt to a live production environment. Unfortunately, I don't have a non-production environment available nor access/permission to create one in the relevant AWS account.
Test your new requirements.txt file in the MWAA local runner. The local runner has specific functionality for testing requirements.txt.
./mwaa-local-env test-requirements
References: