I have multiple pipelines in Azure Data factory that get data from APIs and then push it to a datalake. I get alerts in case one of the pipelines fail. I then go to the ADF instance and rerun the the failed pipeline manually. I am trying to come up with an automated way of rerunning a pipeline in case it fails. Any suggestions or guidance would be helpful. thought of Azure logic apps or powerautomate but turns out don't have the right actions in there to trigger a failed pipeline.

CodePudding user response：

As of now there is no inbuilt method to automate the process of "rerunning from failed activity" in the ADF, but each activity has a Retry option that you should certainly employ. In the pipeline, you may attempt any action as many times as necessary if it fails.

Allow the trigger to point to a new pipeline with a Execute activity that points to the current Azure Datafactory with the copy activity:

Then choose the Advanced -> Wait for completion option. After the execute pipeline is complete, the webhook action should contain logic to halt the DW.

CodePudding user response：

If the pipeline design could be modified then a method can be to

1. Set parameter pMax_rerun_count ( This is to ensure pipeline doesn go into indefinite loop )
2. set 2 variables: (2.a) Pipeline_status default value : Fail (2.b) Max_loop_count default value : 0 ; This would be to ensure the pipeline doesnt run in loops . The value could be set during the pipeline run to have the maximum permissible retry count (i.e. pMax_rerun_count) passed as parameter in the pipeline
3. All activities should be inside and Untill activity which will have expression or(equals(Pipeline_status,'Success'),equals(pMax_rerun_count,Max_loop_count)
4. The first activity inside Untill activity will be Set Variable activity that increment the value of variable Max_loop_count by 1 .
5. The final activity insisde Untill activity will be to Set variable activity that sets Pipeline_status to "Success"

The purpose here is to run all intended activities inside untill block untill the intended activities in pipeline completes successfully . pMax_rerun_count is to ensure pipeline doesnt go into indefinite loops.

This setup can considered as a framework if all pipelines needs to rerun in case of failure