Import library not found in databricks notebook

Question

using Azure Devops pipeline task, I'm importing azure.databricks.cicd.tools library and installing azure-identity and azure-keyvault-secrets. These libraries are installed fine on to a cluster when I add it to a cluster using a bearer token and cluster id however when I run the notebook it says Import module not found. Can you help me where I'm going wrong please?

    - task: AzurePowerShell@5
      inputs:
        azureSubscription: xxxxx
        ScriptType: InlineScript
        Inline: |
          Install-Module -Name azure.databricks.cicd.tools -Force -Scope CurrentUser
        azurePowerShellVersion: LatestVersion   

    - task: AzurePowerShell@5
      inputs:
        azureSubscription: xxxxx
        ScriptType: InlineScript
        Inline: |
          Import-Module -Name azure.databricks.cicd.tools
          Add-DatabricksLibrary -BearerToken $(az-bearer-token) -Region $(az-region) -LibraryType "pypi" -LibrarySettings 'azure-identity' -ClusterId 'xxxxxx'
          Add-DatabricksLibrary -BearerToken $(az-bearer-token) -Region $(az-region) -LibraryType "pypi" -LibrarySettings 'azure.keyvault.secrets' -ClusterId 'xxxxx'
        azurePowerShellVersion: LatestVersion

followed by .....

   - task: configuredatabricks@0
   - task: DataThirstLtd.databricksDeployScriptsTasks.databricksDeployScriptsTask.databricksDeployScripts@0
   - task: executenotebook@0

Databricks notebook:
from azure.identity import ClientSecretCredential
from azure.keyvault.secrets import SecretClient
credential = ClientSecretCredential(directory_id, sp_client_id, sp_client_secret, 'login.microsoftonline.com')

Error message:

ImportError: No module named azure.identity

score 0 · Answer 1 · answered Nov 13 '21 at 14:51

Installation of libraries isn't synchronous - when you execute Add-DatabricksLibrary command it calls install endpoint of Libraries REST API, and documentation says:

The installation is asynchronous - it completes in the background after the request.

You may use the Cluster status REST API to check if all libraries have the INSTALLED status, and trigger the notebook only after that. (I don't see similar functionality in the azure.databricks.cicd.tools library)

Import library not found in databricks notebook

1 Answers1