0

I am trying to configure a devops pipeline to get Databricks workspace to automatically pull the latest version of the repo when master branch is updated.

I am able to generate the DB_PAT but I cannot seem to get this into the DATABRICKS_TOKEN variable properly hence the databricks repos update fails with the following error:

Error: JSONDecodeError: Expecting value: line 1 column 1 (char 0)
##[debug]Exit code 1 received from tool '/usr/bin/bash'
##[debug]STDIO streams have closed for tool '/usr/bin/bash'
##[error]Bash exited with code '1'.

This is my YAML file that I ran in Azure Devops.

 trigger:
- master

jobs:
- job: PullReposUpdate
  
  steps:
  
  - task: AzureCLI@2
    name: DbTokenGen
    inputs:
      azureSubscription: $(azureSubscription)
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        ADBWORKSPACENAME="adb-$(clientName)-$(stageName)"
        echo "$ADBWORKSPACENAME"
        TENANT_ID=$(az account show --query TENANT_ID --output tsv)
        WORKSPACE_ID=$(az resource show --resource-type Microsoft.Databricks/workspaces --resource-group $RGNAME --name $ADBWORKSPACENAME --query id --output tsv)
        TOKEN=$(az account get-access-token --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d | jq --raw-output '.accessToken')
        AZ_TOKEN=$(az account get-access-token --resource https://management.core.windows.net/ | jq --raw-output '.accessToken')
        DB_PAT=$(curl --silent https://$(region).azuredatabricks.net/api/2.0/token/create \
          --header "Authorization: Bearer $TOKEN" \
          --header "X-Databricks-Azure-SP-Management-Token:$AZ_TOKEN" \
          --header "X-Databricks-Azure-Workspace-Resource-Id:$WORKSPACE_ID" \
          --data '{ "lifetime_seconds": 1200, "comment": "Azure DevOps pipeline" }' \
          | jq --raw-output '.token_value')
        echo "##vso[task.setvariable variable=DB_PAT]$DB_PAT"
      failOnStandardError: true
    displayName: 'Generate Token for Databricks'

  - script: |
      pip install --upgrade databricks-cli
    displayName: 'Install dependencies'

  - script: |
      databricks repos update --path $(STAGING_DIRECTORY) --branch "$(branchName)"
    env:
      DATABRICKS_HOST: $(databricks-url)
      DATABRICKS_TOKEN: $(DbTokenGen.DB_PAT)
    displayName: 'Update Production Databricks Repo'

How can I pass the DB_PAT successfully to the databricks cli environment? any suggestion is highly appreciated.

Thank you.

soiryk139
  • 51
  • 6

0 Answers0