3

Can we use managed identities with databricks? What I'm actually trying to achieve is, I have a cluster in databricks, I want it to be able to access secrets or keys stored in an azure key vault.
We generally perform this with VM, by enabling the managed identity and allowing that identity via access policy or Role-based access policy(RBAC) in key vaults.

Can we leverage the concept of manged identities in a similar way with databricks as well? Or is there any other way possible which I can use to access the secrets in key vault from databricks clusters?

P.S. The secret accessed in key vault will be used in init script of the databricks cluster, to perform decrypt opertations.

Sandip Kumar
  • 241
  • 3
  • 25
  • Recently the [Unity Catalog](https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/) has reached GA in Azure. Haven't found a complete solution; however, there's https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/azure-managed-identities to start with. – treedust Sep 16 '22 at 22:17

1 Answers1

1

Managed identity in Azure Databricks isn't supported yet. But right now you can pass the value of secret as an environment variable, and it will be available in your init script - just specify in cluster configuration:

MY_PASSWORD={{secrets/scope/key}}

and then use in the init script:

if [ -n "$MY_PASSWORD" ]; then
  use password
else
  exit 1
fi
Alex Ott
  • 80,552
  • 8
  • 87
  • 132
  • Yeah, I had tried this.. Not a good approach.. It exposes your secret in the web terminal. – Sandip Kumar Aug 26 '21 at 10:27
  • You can disable web terminal on cluster level. Why do you need web terminal? – Alex Ott Aug 26 '21 at 10:38
  • Yeah, I can. But we provide products based on databricks to other customers, we can request them to disable this web terminal but I'm not sure they all will agree to this. – Sandip Kumar Aug 27 '21 at 13:13