I have an Airflow job with 77 (11*7) Task. When any one Task fails, I do not want the DAG to FAIL. Instead, I want to auto restart the failed Task again (twice) and then even if it fails, I need the DAG to fail. Is is possible to auto restart the Task? Task failure happens when i copy a file from SFTP to S3. The file size is 4GB. Due to some timeout error (in AWS), this failure happens. My ASK: Can anyone please help me to figure out on How to restart a Task automatically in case of failure. Also how to restart a DAG automatically in case of failure. This restart should happen twice and then FAIL the TASK/DAG.

CodePudding user response：

You just need to provide retries arguments which is the number of retries that should be performed before failing the task, by default it's equal to the setting core.default_task_retries if it's provided or 0 if not. (you can provide it to the dag default args if you want to set it for all the dag tasks)

You can also decide how long Airflow has to wait before retry the task by setting retry_delay, and if you want an exponential wait time, you can set retry_exponential_backoff to True.