Using databricks workspace in the same configuration as the databricks provider

Question

I'm having some trouble getting the azurerm & databricks provider to work together.

With the azurerm provider, setup my workspace

resource "azurerm_databricks_workspace" "ws" {
  name                        = var.workspace_name
  resource_group_name         = azurerm_resource_group.rg.name
  location                    = azurerm_resource_group.rg.location
  sku                         = "premium"
  managed_resource_group_name = "${azurerm_resource_group.rg.name}-mng-rg"
  custom_parameters {
    virtual_network_id  = data.azurerm_virtual_network.vnet.id
    public_subnet_name  = var.public_subnet
    private_subnet_name = var.private_subnet
  }
}

No matter how I structure this, I can't say seem to get the azurerm_databricks_workspace.ws.id to work in the provider statement for databricks in the the same configuration. If it did work, the above workspace would be defined in the same configuration and I'd have a provider statement that looks like this:

provider "databricks" {
  azure_workspace_resource_id = azurerm_databricks_workspace.ws.id
}

Error:

I have my ARM_* environment variables set to identify as a Service Principal with Contributor on the subscription.

I've tried in the same configuration & in a module and consuming outputs. The only way I can get it to work is by running one configuration for the workspace and a second configuration to consume the workspace.

This is super suboptimal in that I have a fair amount of repeating values across those configurations and it would be ideal just to have one.

Has anyone been able to do this? Thank you :)

Thinking about your response makes me think I've stated something incorrectly. I'll start generating some errors. Probably PEBKAC ;) — Josh Robinson, May 10 '21 at 11:26
It’s also easier to open an issue in repository itself. It’s better monitored — Alex Ott, May 10 '21 at 11:44

score 4 · Answer 1 · answered May 11 '21 at 09:28

I've had the exact same issue with a not working databricks provider because I was working with modules. I separated the databricks infra (Azure) with databricks application (databricks provider).

In my databricks module I added the following code at the top, otherwise it would use my azure setup:

terraform {
  required_providers {
    databricks = {
      source = "databrickslabs/databricks"
      version = "0.3.1"
    }
  }
}

In my normal provider setup I have the following settings for databricks:

provider "databricks" {
  azure_workspace_resource_id = module.databricks_infra.databricks_workspace_id
  azure_client_id             = var.ARM_CLIENT_ID
  azure_client_secret         = var.ARM_CLIENT_SECRET
  azure_tenant_id             = var.ARM_TENANT_ID
}

And of course I have the azure one. Let me know if it worked :)

ok, this follows an approach I was using as well. I have the workspace living in a module in one of my experiment branches. The difference between mine and yours is your databricks provider setup. I was just setting the `azure_workspace_resource_id`, but I'm not even sure that I knew you could do this with the ARM* variables! Thank you! I'll give it a try and if it works, you can have the answer :) — Josh Robinson, May 11 '21 at 13:10
@JoshRobinson Did it work? If you need any more code/help, let me know. — 54m, May 12 '21 at 13:22

score 1 · Answer 2 · answered May 15 '21 at 13:28

If you experience technical difficulties with rolling out resources in this example, please make sure that environment variables don't conflict with other provider block attributes. When in doubt, please run TF_LOG=DEBUG terraform apply to enable debug mode through the TF_LOG environment variable. Look specifically for Explicit and implicit attributes lines, that should indicate authentication attributes used. The other common reason for technical difficulties might be related to missing alias attribute in provider "databricks" {} blocks or provider attribute in resource "databricks_..." {} blocks. Please make sure to read alias: Multiple Provider Configurations documentation article.

score 0 · Answer 3 · answered May 11 '21 at 03:48

0

From the error message, it looks like Authentication is not configured for provider could you please configure it through the one of following options mentioned above.

For more details, refer Databricks provider - Authentication.

For passing the custom_parameters, you may checkout the SO thread which addressing the similar issue.

In case if you need more help on this issue, I would suggest to open an issue here: https://github.com/terraform-providers/terraform-provider-azurerm/issues

answered May 11 '21 at 03:48

CHEEKATLAPRADEEP

12,191
1
19
42

I have all of my ARM_* environment variables set. I don't actually have to do what Sam is doing to get something working. Usually setting the variables and adding the host to the provider config does the trick. This is what I expect "AZ authentication" means in the error message. However, the error message isn't specific enough. – Josh Robinson May 12 '21 at 11:16

Using databricks workspace in the same configuration as the databricks provider

3 Answers3