I am trying to configure a devops pipeline to get Databricks workspace to automatically pull the latest version of the repo when master branch is updated.
I am able to generate the DB_PAT but I cannot seem to get this into the DATABRICKS_TOKEN variable properly hence the databricks repos update fails with the following error:
Error: JSONDecodeError: Expecting value: line 1 column 1 (char 0)
##[debug]Exit code 1 received from tool '/usr/bin/bash'
##[debug]STDIO streams have closed for tool '/usr/bin/bash'
##[error]Bash exited with code '1'.
This is my YAML file that I ran in Azure Devops.
trigger:
- master
jobs:
- job: PullReposUpdate
steps:
- task: AzureCLI@2
name: DbTokenGen
inputs:
azureSubscription: $(azureSubscription)
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
ADBWORKSPACENAME="adb-$(clientName)-$(stageName)"
echo "$ADBWORKSPACENAME"
TENANT_ID=$(az account show --query TENANT_ID --output tsv)
WORKSPACE_ID=$(az resource show --resource-type Microsoft.Databricks/workspaces --resource-group $RGNAME --name $ADBWORKSPACENAME --query id --output tsv)
TOKEN=$(az account get-access-token --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d | jq --raw-output '.accessToken')
AZ_TOKEN=$(az account get-access-token --resource https://management.core.windows.net/ | jq --raw-output '.accessToken')
DB_PAT=$(curl --silent https://$(region).azuredatabricks.net/api/2.0/token/create \
--header "Authorization: Bearer $TOKEN" \
--header "X-Databricks-Azure-SP-Management-Token:$AZ_TOKEN" \
--header "X-Databricks-Azure-Workspace-Resource-Id:$WORKSPACE_ID" \
--data '{ "lifetime_seconds": 1200, "comment": "Azure DevOps pipeline" }' \
| jq --raw-output '.token_value')
echo "##vso[task.setvariable variable=DB_PAT]$DB_PAT"
failOnStandardError: true
displayName: 'Generate Token for Databricks'
- script: |
pip install --upgrade databricks-cli
displayName: 'Install dependencies'
- script: |
databricks repos update --path $(STAGING_DIRECTORY) --branch "$(branchName)"
env:
DATABRICKS_HOST: $(databricks-url)
DATABRICKS_TOKEN: $(DbTokenGen.DB_PAT)
displayName: 'Update Production Databricks Repo'
How can I pass the DB_PAT successfully to the databricks cli environment? any suggestion is highly appreciated.
Thank you.