Azure Databricks workspace using terraform

Question

Trying to create Databricks workspace using terraform but unsupported arguments:

resource "azurerm_databricks_workspace" "workspace" {
  name                = "testdata"
  resource_group_name = "cloud-terraform"
  location            = "east us"
  sku                 = "premium"
  virtual_network_id  = azurerm_virtual_network.vnet.id
  public_subnet_name  = "databrickpublicsubnet"
  public_subnet_cidr  = "10.0.0.0/22"
  private_subnet_name = "databrickprivatesubnet"
  private_subnet_cidr  = "10.0.0.0/22"
    
  tags = {
    Environment = "terraformtest"
  }
}

Error: An argument named "virtual_network_id" is not expected here. An argument named "public_subnet_name" is not expected here. An argument named "public_subnet_cidr" is not expected here.

score 2 · Accepted Answer · answered Jul 23 '20 at 00:08

2

I haven't tried to set up databricks via Terraform, but I believe (per the docs) you need add those properties in a block:

resource "azurerm_databricks_workspace" "workspace" {
  name                = "testdata"
  resource_group_name = "cloud-terraform"
  location            = "east us"
  sku                 = "premium"
  
  custom_parameters {
    virtual_network_id  = azurerm_virtual_network.vnet.id
    public_subnet_name  = "databrickpublicsubnet"
    private_subnet_name = "databrickprivatesubnet"
  }

  tags = {
    Environment = "terraformtest"
  }
}

The two cidr entries aren't part of the TF documentation.

answered Jul 23 '20 at 00:08

mherzig

1,528
14
26

for databricks to be created in a vnet, 2 new subnets are deployed. As per the document terraform does not have 2 cidr entries are argument parameters as of now.. – mikeknows Jul 23 '20 at 06:11
True. Although the ARM template docs (https://learn.microsoft.com/en-us/azure/templates/microsoft.databricks/workspaces) don't specify the IP blocks either, and that's what TF is creating. I wonder if you would add the subnets to the VNET first, which would include the IP block, and then all you need is the name of the subnet. The portal could just be handling that in one step. – mherzig Jul 23 '20 at 06:24

score 1 · Answer 2 · answered Mar 10 '22 at 20:06

true. you can add terraform commands to create the subnets (assuming vnet already exists, you can use data azurerm_virtual_network then create the two new subnets, then reference the names of the two new public/private subnets.

Then you run into what seems to be a chicken/egg issue though.

You get Error: you must define a value for 'public_subnet_network_security_group_association_id' if 'public_subnet_name' is set.

Problem is, the network security group is typically auto-generated on creation of the databrick workspace (like databricksnsgrandomstring), which works when creating it in the portal, but via terraform, I have to define it to create the workspace, but it doesn't yet exist until I create the workspace. The fix is to not let it generate it's own nsg name, but name it yourself with an nsg resource block.

below is code I use (dbname means databricks name!). here I'm adding to an existing resource group 'qa' and existing vnet as well, only showing the public subnet and nsg association, you can easily add the private ones). just copy/modify in your own tf file(s). and you'll definitely need to change the address_prefix to your own CIDR values that works within your vnet and not stomp on existing subnets within.

resource "azurerm_subnet" "public" {
    name = "${var.dbname}-public-subnet"
    resource_group_name = data.azurerm_resource_group.qa.name
    virtual_network_name = data.azurerm_virtual_network.vnet.name
    address_prefixes = ["1.2.3.4/24"]

    delegation {
        name = "databricks_public"
        service_delegation {
            name = "Microsoft.Databricks/workspaces"
        }
    }
}

resource "azurerm_network_security_group" "nsg" {
    name = "${var.dbname}-qa-databricks-nsg"
    resource_group_name = data.azurerm_resource_group.qa.name
    location= data.azurerm_resource_group.qa.location
}

resource "azurerm_subnet_network_security_group_association" "nsga_public" {
    network_security_group_id = azurerm_network_security_group.nsg.id
    subnet_id = azurerm_subnet.public.id
}

Then in your databricks_workspace block, replace your custom parameters with

    custom_parameters {
        public_subnet_name  = azurerm_subnet.public.name
        public_subnet_network_security_group_association_id = azurerm_subnet_network_security_group_association.nsga_public.id
        private_subnet_name = azurerm_subnet.private.name
        private_subnet_network_security_group_association_id = azurerm_subnet_network_security_group_association.nsga_private.id
        virtual_network_id  = data.azurerm_virtual_network.vnet.id
    }

Azure Databricks workspace using terraform

2 Answers2

Linked