How to set up unity catalog access connector with terraform

Question

I am trying to update our Databricks account and workspace to use Unity Catalog, we've got all our infrastructure templated using terraform, I'm trying to deploy an azapi resource as detailed in the terraform unity catalog upgrade docs but am not having any luck. My terraform code below is used to create the connector

  provider "azuread" {
  client_id     = var.client_id
  client_secret = var.client_secret
  tenant_id     = var.tenant_id
}
provider "azurerm" {
  features {}
  client_id       = var.client_id
  client_secret   = var.client_secret
  subscription_id = var.subscription_id
  tenant_id       = var.tenant_id
}
provider "azapi" {
  client_id       = var.client_id
  tenant_id       = var.tenant_id
  subscription_id = var.subscription_id
  client_secret   = var.client_secret
}
resource "azurerm_resource_group" "this" {
  name     = "${local.prefix}-metaverse-migration-rg"
  location = var.region
  tags     = local.tags
}

resource "azapi_resource" "access_connector" {
type      = "Microsoft.Databricks/accessConnectors@2022-04-01-preview"
name      = "${local.prefix}-databricks-mi"
location  = azurerm_resource_group.this.location
parent_id = azurerm_resource_group.this.id
identity {
  type = "SystemAssigned"
}
body = jsonencode({
  properties = {}
})
}

the error message I receive when running this on terraform cloud is

Error: creating/updating "Resource: (ResourceId \"/subscriptions/mysubguid/resourceGroups/databricks-metaverse-migration-rg/providers/Microsoft.Databricks/accessConnectors/databricks-databricks-mi\" / Api Version \"2022-04-01-preview\")": PUT https://management.azure.com/subscriptions/mysubguid/resourceGroups/databricks-metaverse-migration-rg/providers/Microsoft.Databricks/accessConnectors/databricks-databricks-mi -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: 403 -------------------------------------------------------------------------------- { "error": { "code": "403", "message": "User not authorized." } } -----------------------------------

I'm running this under a service principal with ownership over the subscription but was getting the same error when it was a contributor. I have a suspicion I need to use a Managed Identity but not sure how/where

Thanks!

score 0 · Answer 1 · answered Oct 14 '22 at 17:18

I think this is a similar issue to the one with datafactory and configuring git via TF. If using a SP you need a custom role. I'd suggest testing the permissions of the SP by logging into azure cli with it and trying to create the connector through CLI or ARM template. This will give you a more detailed message of what's the problem.

https://learn.microsoft.com/en-us/azure/templates/microsoft.databricks/accessconnectors?pivots=deployment-language-arm-template

Azure RBAC role 'ConfigureFactoryRepo' is required to be granted over the entire subscription to Terraform Service Principal to spin up Data Factory

Thanks for the tips, turns out I needed to assign the IdentityProvider.ReadWrite.All to my service principal — Compsci is math base 2, Oct 19 '22 at 01:01

score 0 · Accepted Answer · answered Oct 18 '22 at 21:43

After adding the below permissions to the service principal, I was able to successfully deploy the access connector. For the access connector itself, I believe only the IdentityProvider.ReadWrite.All permission is needed with subsequent permissions being used for other deployment aspects

How to set up unity catalog access connector with terraform

2 Answers2