I created a spark job with parameters and attached an interactive cluster to it in Azure Databricks. I have a pipeline in Azure data factory using a notebook activity. When i created How a linked service, it is pointing to my interactive cluster. How do I use ADF to run pipeline using the job I creates in databricks? Thanks

CodePudding user response：

You can execute the databricks job with Notebook activity in ADF. But as far as I know, the Notebook activity is used for only to create and execute the new databricks job for the notebook.

I created a spark job with parameters

So, you have already created a databricks job.

To execute the existing job of databricks in ADF, you can use other ADF activities and Databricks jobs API.

Please go through this Microsoft Official blog by Leo Furlong and ClintonWFord which has step by step explanation to run a databricks job in ADF involving the activities like web activity for Run now API and wait actvity.

With this approach you can execute whichever job you want by passing the JobId and Databricks workspace Id in pipeline parameters as shown in the above link.

If you want to execute multiple jobs in ADF you can try this approach by Iván Gómez Arnedo.