1

I am trying to call datadog-install-driver-workers.sh using terraform resource databricks_global_init_script and this script required 2 input values to pass DD_API_KEY and DD_ENV How do I pass these these values along with source script path?

resource "databricks_global_init_script" "init1" {
  source = "${path.module}/datadog-install-driver-workers.sh"
  name   = "my init script"
}
Cyrus
  • 84,225
  • 14
  • 89
  • 153
raj
  • 137
  • 1
  • 1
  • 10

2 Answers2

1

I would go a different route - instead of the requiring each job & cluster to specify necessary values, you can use the templatefile function to substitute necessary values in the script, like this:

locals {
  script_path = "${path.module}/datadog-install-driver-workers.sh"
  params = {
    DD_ENV = "dev"
    DD_API_KEY = "aaaaa"
  }
}

resource "databricks_global_init_script" "init" {
  name = "datadog script"
  content_base64 = base64encode(templatefile(local.script_path, local.params))
}

with the script template as following:

#!/bin/bash
#


DD_ENV="${DD_ENV}"
DD_API_KEY="${DD_API_KEY}"

echo "Some code that outputs $${DD_ENV}"

and this will generate it correctly:

enter image description here

The only thing that you need to take into account is that you may need to escape shell variables substitutions that use the same syntax as Terraform: ${var} to $${var} - see documentation.

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
0

Init scripts do not take any parameters. Your code is fine as is.

Datadog docs mention that DD_API_KEY and DD_ENV are expected to be set as environment variables. Those can be defined during cluster creation. Using UI, this is done on Advances Options -> Spark -> Environment variables (docs). Also both Cluster API and Terraform databricks_cluster resource support spark_env_vars parameter that you can use. For example, with Cluster API this would be the relevant payload:

{
  "cluster_name": "my-cluster",
  "spark_env_vars": {
    "DD_API_KEY": "blahblah",
    "DD_ENV": "blah"
  },
  [...other attributes...]
}

Note that you could enforce that specific environment variables are always defined using cluster policies.

Kombajn zbożowy
  • 8,755
  • 3
  • 28
  • 60
  • cluster policies doesn't play well with terraform resources – Alex Ott May 16 '23 at 08:45
  • @AlexOtt Can you explain a bit more? – Kombajn zbożowy May 16 '23 at 09:44
  • 1
    Cluster policy injects default values into Cluster/Job configuration, but when Terraform provider reads corresponding resources and finds values that aren't set on the resource level, so it identifies them as changes, and attempts to change. As result, each plan/apply will lead to changes... – Alex Ott May 16 '23 at 10:17
  • Can you please share recommended way to use datadog init script? I m using terraform to create workspace/cluster. Also, when added datadog-init as part of global init configuration I observed on job cluster after the script finishes it cancels any pending tasks which causes the Job to be cancelled. How do we avoid this situation? thanks – raj May 16 '23 at 13:37