1

Hoping to get some clarity on the Azure DataBricks (ADB) integration with Azure Key Vault (AKV).

1- When ADB is configured with AKV backed secret which component connects to the AKV? The control plane or the cluster (data plane) ? (I'm assuming it's control plane but not sure...)

2- If AKV is configured with Private link/endpoint and ADB cluster in deployed in a VNET ( i.e. vnet injected) it is using secure cluster connectivity option (i.e. No PIP) then will the AKV backed secret suppose to work in this deployment? (If the answer to # 1 is control plane then I don't see how this will work...?)

Thanks.

csaif7
  • 55
  • 1
  • 8

1 Answers1

0

• When Azure Databricks is configured with Azure Key Vault backed secret, it is the control plane that connects to the Azure Key Vault as the notebook which calls the azure key vault stored secret scope is stored in the Azure databricks workspace since it is managed service data encrypted at rest with a Databricks-managed key.

• Also, when you configure Azure key vault by integrating it with azure private link, it will configure a private zone dns by integrating it with the private endpoint that will be created. And since, traffic between the azure key vault virtual network and the azure databricks cluster traverses over the Microsoft backbone network, it eliminates exposure from the public Internet.

• The Azure databricks cluster will be having a resource ID when it is configured and that can be selected when assigning a private endpoint in aure key vault while configuring with azure private link thus successfully importing the azure key vault backed secrets in the azure databricks cluster over a private network.

• Please find the below images for more clarity: -

Secure cluster connectivity

Please find the below links for more information: -

https://learn.microsoft.com/en-us/azure/databricks/security/keys/customer-managed-key-managed-services-azure

https://learn.microsoft.com/en-us/azure/databricks/security/secure-cluster-connectivity

https://learn.microsoft.com/en-us/azure/key-vault/general/private-link-service?tabs=portal

Kartik Bhiwapurkar
  • 4,550
  • 2
  • 4
  • 9
  • Thank you for your response. If it's control plane that communicate with AKV then how will the control plan resolve the dns for AKV using the private dns that is configured and hosted in customer managed subscription? Unless something links the control plane vnet to the private dns zone it will use the public dns and PIP for AKV not the private link. Unless I'm missing something here, I don't see AKV with private link in this configuration to work - the alternate will be to block internet access on AKV and enable microsoft trusted services access. – csaif7 Sep 22 '21 at 12:25