1

I am trying to create data toolchain in automated way. I am using Azure, databricks-cli. https://github.com/Azure-Samples/modern-data-warehouse-dataops/blob/main/e2e_samples/parking_sensors/scripts/deploy_infrastructure.sh

I have issue to authenticate to databrics with aad token, generate PAT.

echo "Generate Databricks token"
databricks_host=https://$(echo "$arm_output" | jq -r '.properties.outputs.databricks_output.value.properties.workspaceUrl')
databricks_workspace_resource_id=$(echo "$arm_output" | jq -r '.properties.outputs.databricks_id.value')
databricks_aad_token=$(az account get-access-token --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d --output json | jq -r .accessToken) # Databricks app global id

This is causing me problems.

# Use AAD token to generate PAT token
databricks_token=$(DATABRICKS_TOKEN=$databricks_aad_token \
    DATABRICKS_HOST=$databricks_host \
    bash -c "databricks tokens create --comment 'deployment'" | jq -r .token_value)

How to authenticate to databricks in order to being able to use databricks cli ??

later I am trying to create secrets but it fails as I am not authenticated.

# Create secret scope
databricks secrets create-scope --scope "$scope_name" \
    --scope-backend-type AZURE_KEYVAULT \
    --resource-id "$KEYVAULT_RESOURCE_ID" \
    --dns-name "$KEYVAULT_DNS_NAME"

Thank you Alex, unfortunately it still does not work on Azure. System:

az login - as subscription owner done

databricks -v
Version 0.16.4

export DATABRICKS_HOST='https://xxx-xxx.16.azuredatabricks.net'
export DATABRICKS_TOKEN=$(az account get-access-token --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d --output json | jq -r .accessToken)

First attempt:

databricks tokens list
Error: b'<html>\n<head>\n<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>\n<title>Error 403 User not authorized.</title>\n</head>\n<body><h2>HTTP ERROR 403</h2>\n<p>Problem accessing /api/2.0/token/list. Reason:\n<pre>    User not authorized.</pre></p>\n</body>\n</html>\n'

Second attempt:

databricks secrets create-scope --scope "XXX"     --scope-backend-type AZURE_KEYVAULT     --resource-id "$KEYVAULT_RESOURCE_ID"     --dns-name "$KEYVAULT_DNS_NAME"
Error: b'<html>\n<head>\n<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>\n<title>Error 403 User not authorized.</title>\n</head>\n<body><h2>HTTP ERROR 403</h2>\n<p>Problem accessing /api/2.0/secrets/scopes/create. Reason:\n<pre>    User not authorized.</pre></p>\n</body>\n</html>\n
aazure
  • 11
  • 1
  • 3

1 Answers1

0

You don't need personal access token to create a scope. Just set DATABRICKS_HOST to URL of workspace and DATABRICKS_TOKEN to value of AAD token, and then use databricks secrets create-scope - this command won't work with personal access token. Something like this:

export DATABRICKS_HOST=...
export DATABRICKS_TOKEN=$(az account get-access-token --resource \
  2ff814a6-3304-4ab8-85cb-cd0e6f879c1d --output tsv --query accessToken)
databricks secrets create-scope --scope "$scope_name" \
    --scope-backend-type AZURE_KEYVAULT \
    --resource-id "$KEYVAULT_RESOURCE_ID" \
    --dns-name "$KEYVAULT_DNS_NAME"

but please note that this AAD token should be of the real user, not service principal - that's a known limitation:

You need an Azure AD user token to create an Azure Key Vault-backed secret scope with the Databricks CLI. You cannot use an Azure Databricks personal access token or an Azure AD application token that belongs to a service principal.

P.S. If you're automating things, you can look onto Databricks Terraform Provider that can help with such stuff.

Alex Ott
  • 80,552
  • 8
  • 87
  • 132