1

I want to be able to call Databricks API from DevOps pipeline. I can do this usint personal access token for my account, however I want to make API calls user independent so I wanted to use Service principal (App registration). I followed this tutorial https://learn.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/aad/service-prin-aad-token to create access token for the service principal, however I have 2 issues:

  1. such generated token expires in 1 hour - is there any elegant was to automatically refresh it?
  2. even when calling the ADB API using this token I get 403 unauthorized - is there anything else I should do? The app registration has Contributor role for the ADB service.

EDIT: Added API Permission for the AzureDatabricks in App registration and Granted admin consent, however still no luck.

romanzdk
  • 930
  • 11
  • 30

2 Answers2

2

So I found 3 possible solutions at the end.

  1. Generate access token for service principal, generate management service token for service principal and use both of these to access Databricks API - reference
  2. Use access token and management token to generate Databricks Personal access token for the service principal using Databricks Token API, then you can use it for Databricks CLI - reference
  3. Authenticate to Databricks via CLI using AAD token (reference and Databricks CLI help):
    1. az login --service-principal -u <app-id> -p <app-password> --tenant <tenant-id>
    2. token_response=$(az account get-access-token --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d)
    3. export DATABRICKS_AAD_TOKEN=$(jq .accessToken -r <<< "$token_response")
    4. databricks configure --host https://<adb-url> --aad-token
romanzdk
  • 930
  • 11
  • 30
1
  1. such generated token expires in 1 hour - is there any elegant was to automatically refresh it?

No, client credentials flow doesn't support refresh token. You can try to get a new token, please refer to this issue.

  1. even when calling the ADB API using this token I get 403 unauthorized - is there anything else I should do? The app registration has User role for the ADB service.

Make sure your service principal have a Contributor role assigned.

There are two kinds of resource in different situations.

  1. API access for service principals that are Azure Databricks workspace users and admins

    resource=2ff814a6-3304-4ab8-85cb-cd0e6f879c1d

  2. API access for service principals that are not workspace users

    resource=https://management.core.windows.net/

unknown
  • 6,778
  • 1
  • 5
  • 14
  • Thank you for you reply. When I use `resource=2ff..` I get `Error 403 User not authorized.` when used with ADB API. When I use `resource=https:..` I get `Error 400 io.jsonwebtoken.IncorrectClaimException: Expected aud claim to be: 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d`. I also checked IAM for ADB service and the service principal has Contributor role. Any idea why is that? – romanzdk Oct 15 '21 at 10:10
  • Did you add Azure Databricks permission in "API Permissions"? Please navigate to API Permissions -> add a permission -> API my organization uses -> search "AzureDatabricks" and add it. Then click grant admin consent. – unknown Oct 18 '21 at 05:58
  • still 403 Unauthorized :( – romanzdk Oct 19 '21 at 13:41
  • Could you let me know the specific API you are calling? – unknown Oct 20 '21 at 01:39
  • Tried list clusters 2.0/clusters/list and jobs run-now. None of them working. – romanzdk Oct 20 '21 at 04:12
  • So, finally, I used management endpoint access token to access the Databricks REST API and now it works, BUT only from my local machine... Not from the DevOps agent.. I already tried microsoft-hosted one as well as self-hosted.. I am getting `Error 400 io.jsonwebtoken.security.SignatureException: JWT signature does not match locally computed signature.` – romanzdk Oct 20 '21 at 15:03