1

Do we have a solution in python to list job runs that are created using Azure VM managed identity in Azure Databricks.

Appreciate the help!

I am getting http 403 error when using managed identity library in python

from azure.identity import ManagedIdentityCredential  
credential = ManagedIdentityCredential() 
# Obtain an access token  
from azure.identity import DefaultAzureCredential  
credentials = DefaultAzureCredential()
access_token = credentials.get_token("management.azure.com/") 
headers = { 'Authorization': 'Bearer ' + access_token,
    'Content-Type': 'application/json' } 
# Set the URL for the Databricks REST API 
endpoint url = "databricks_url" + '/api/2.0/clusters/list' 
# Make the REST API call to the Databricks endpoint 
response = requests.get(url, headers=headers)  
print(response.json())
Alex Ott
  • 80,552
  • 8
  • 87
  • 132
user3651363
  • 11
  • 2

1 Answers1

0

If managed identity isn't added into the Databricks workspace yet, then having only access token isn't enough - you also need to provide an additional access token for accessing Azure management API (the https://management.core.windows.net/ resource URL), and it should be provided as the X-Databricks-Azure-SP-Management-Token header, together with Databricks Workspace Resource ID as the X-Databricks-Azure-Workspace-Resource-Id.

And then you need to generate the access token to the for Databricks workspace resource (2ff814a6-3304-4ab8-85cb-cd0e6f879c1d) that should be sent as bearer token.

This specific scenario is described in the details the official documentation.

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
  • Managed Identity is already added to the workspace.I cannot follow the documentation since I don't have service principal. – user3651363 Feb 01 '23 at 16:20
  • If it’s already added, then your solution is 2nd paragraph. You were generating AAD token for the wrong scope - see above in the same doc. (MI behaves like SP, you simply don’t have secret - you still generate tokens for a specific resource). – Alex Ott Feb 01 '23 at 19:05
  • Here is an example for service principal, but you can see the correct scope: https://github.com/alexott/databricks-playground/blob/main/dbsql-with-aad-token/dbsql-with-aad-token.py#L17 – Alex Ott Feb 01 '23 at 19:07
  • Thanks I was able to resolve with scope. – user3651363 Feb 03 '23 at 13:28