Azure Databricks API to create job, job doesn't get created after successful call to the API-CodePudding

I am using python 3.6 to make API calls to Azure Databricks to create a job to run a specific notebook. I have followed the instruction of using the API at

However, when I look into list of jobs, no new job is created despite the 200 status code received! You can see below the job I have created is not there.

I also changed the API endpoint from azuredatabricks.net//2.0/jobs/create to azuredatabricks.net//2.1/jobs/create, still I get successful run but no job is being created! I can't understand what I am doing wrong. And if I am doing something wrong, how come it doesn't raise exception and gives me 200 status code.

One final point to be able to regenerate the problem I am facing: To get the above two variables for DBRKS_BEARER_TOKEN and DBRKS_MANAGEMENT_TOKEN, you can run the following script and manually replace os.environ['DBRKS_BEARER_TOKEN'] and os.environ['DBRKS_MANAGEMENT_TOKEN'] with the printed values after script execution:

import requests
import json
import os


TOKEN_BASE_URL = 'https://login.microsoftonline.com/'   os.environ['SVCDirectoryID']   '/oauth2/token'
TOKEN_REQ_HEADERS = {'Content-Type': 'application/x-www-form-urlencoded'}
TOKEN_REQ_BODY = {
       'grant_type': 'client_credentials',
       'client_id': os.environ['SVCApplicationID'],
       'client_secret': os.environ['SVCSecretKey']}



def dbrks_management_token():
        TOKEN_REQ_BODY['resource'] = 'https://management.core.windows.net/'
        response = requests.get(TOKEN_BASE_URL, headers=TOKEN_REQ_HEADERS, data=TOKEN_REQ_BODY)
        if response.status_code == 200:
            print(response.status_code)
        else:
            raise Exception(response.text)
        return response.json()['access_token']


def dbrks_bearer_token():
        TOKEN_REQ_BODY['resource'] = '2ff814a6-3304-4ab8-85cb-cd0e6f879c1d'
        response = requests.get(TOKEN_BASE_URL, headers=TOKEN_REQ_HEADERS, data=TOKEN_REQ_BODY)
        if response.status_code == 200:
            print(response.status_code)
        else:
            raise Exception(response.text)
        return response.json()['access_token']

DBRKS_BEARER_TOKEN = dbrks_bearer_token()
DBRKS_MANAGEMENT_TOKEN = dbrks_management_token()

os.environ['DBRKS_BEARER_TOKEN'] = DBRKS_BEARER_TOKEN 
os.environ['DBRKS_MANAGEMENT_TOKEN'] = DBRKS_MANAGEMENT_TOKEN 

print("DBRKS_BEARER_TOKEN",os.environ['DBRKS_BEARER_TOKEN'])
print("DBRKS_MANAGEMENT_TOKEN",os.environ['DBRKS_MANAGEMENT_TOKEN'])

SVCDirectoryID is Azure Active Directory (AAD) service principal tenant Id
SVCApplicationID is the value of AAD service principal client Id.
SVCSecretKey is AAD service principal secret key.

Thank you for your valuable input.

CodePudding user response：

You're mixing up the API versions - the tasks array could be used only with Jobs API 2.1, but you're using Jobs API 2.0. Another error is that you have // between host name & path.

Just change dbrks_create_job_url to "https://" os.environ['DBRKS_INSTANCE'] ".azuredatabricks.net/api/2.1/jobs/create"