I want to export my runtime duration of the each pipeline to CSV. How can this be done or achieved? The below is scheduled Pipeline which run to end, i can use the the Export CSV button to get duration of these pipelines but each execute pipeline got so many individual jobs(In this example will take P_Weekly_RV_Load_1) and how can I get those details of all those individual jobs run in p_weekly_RV_Load_! to csv

CodePudding user response：

Navigate to the Monitoring view in ADF, click on "Pipeline runs" and then "Debug". On the top right you'll see "Export to CSV".

CodePudding user response：

When you say individual jobs I am assuming you mean the activity runs inside of a pipeline. Currently activity level detail is not exported to CSV, just the pipeline details.

To capture activity level detail there are a couple of options:

1. Use diagnostics settings and send the Activity Runs to storage, Log Analytics, or Event Hubs. See https://docs.microsoft.com/en-us/azure/data-factory/monitor-configure-diagnostics/ on how to set this up.
2. Use the SDKs via REST, Powershell, etc. to query the activity run endpoints and export the data programmatically. See https://docs.microsoft.com/en-us/rest/api/datafactory/activity-runs/query-by-pipeline-run as an example.