How to automate the bitbucket repository to databricks repos by using ci/cd pipeline

Question

Can someone help on this how to automate the ci/cd pipeline to update and create new files in databricks repos from bitbucket repositories..

Answer 1

If you want to sync changes from BitBucket repository into Databricks Repos, then you have following possibilities:

• Use databricks repos update command of the databricks-cli package, like this:

databricks repos update --path /Repos/user/repository --branch <branch_name>

• Use Update command of Repos API, but it's too low-level because it doesn't work with paths, and you need to know Repository ID that could obtained via Workspace API. So it's several commands instead of single one:

curl -s -n -X GET -o /tmp/staging-repo-info.json "$DATABRICKS_HOST/api/2.0/workspace/get-status" -H "Authorization: Bearer $DATABRICKS_TOKEN" -d '{"path":"/Repos/Staging/databricks-nutter-projects-demo"}'
export STAGING_REPOS_ID=$(cat /tmp/staging-repo-info.json|grep '"object_type":"REPO"'|sed -e 's|^.*"object_id":\([0-9]*\).*$|\1|')
curl -s -n -X PATCH -o "/tmp/$(Build.SourceBranchName)-out.json" "$DATABRICKS_HOST/api/2.0/repos/$STAGING_REPOS_ID" \
    -H "Authorization: Bearer $DATABRICKS_TOKEN" -d "{\"branch\": \"$(Build.SourceBranchName)\"}"

P.S. You can find end-to-end CI/CD demo with Repos and Azure DevOps in this repository. Although it's not a BitBucket, but the structure of the pipeline remains the same.