How can I automatically add a kubernetes client secret as a file mount in Terraform?

Question

I am setting up External-DNS with Terraform. Per the documentation, I have to manually create an azure.json file and mount it as a secret volume. The directions also state:

The Azure DNS provider expects, by default, that the configuration file is at /etc/kubernetes/azure.json

{
  "tenantId": "01234abc-de56-ff78-abc1-234567890def",
  "subscriptionId": "01234abc-de56-ff78-abc1-234567890def",
  "resourceGroup": "MyDnsResourceGroup",
  "aadClientId": "01234abc-de56-ff78-abc1-234567890def",
  "aadClientSecret": "uKiuXeiwui4jo9quae9o"
}

I then run kubectl create secret generic azure-config-file --from-file=/local/path/to/azure.json to mount the secret as a file.

The problem is that those values are dynamic, and I need to do this automatically per a CI/CD pipeline. I'm using Terraform Kubernetes resources, and here I've used the kubernetes_secret resource.

resource "kubernetes_secret" "azure_config_file" {
  metadata {
    name = "azure-config-file"
  }

  data = {
    tenantId = data.azurerm_subscription.current.tenant_id
    subscriptionId = data.azurerm_subscription.current.subscription_id
    resourceGroup = azurerm_resource_group.k8s.name
    aadClientId = azuread_application.sp_externaldns_connect_to_dns_zone.application_id
    aadClientSecret = azuread_application_password.sp_externaldns_connect_to_dns_zone.value
  }

  depends_on = [
    kubernetes_namespace.external_dns,
  ]
}

The secret gets mounted, but the pod never sees it and it results in a crashLoopBackoff. This may not be the best direction.

How do I automate this process with Terraform and get it mounted correctly?

For reference, this is the related section of the YAML manifest

...

       volumeMounts:
        - name: azure-config-file
          mountPath: /etc/kubernetes
          readOnly: true
      volumes:
      - name: azure-config-file
        secret:
          secretName: azure-config-file
          items:
          - key: externaldns-config.json
            path: azure.json

In the documentation the `mountPath` is saying the path has to be quoted, so I would start from there. The question I have is: should the secret name be `azure.json` or `externaldns-config.json`? If it should be `azure.json`, then you've reversed the order for `key` and `path`. — Marko E, Feb 12 '22 at 21:33

score 1 · Answer 1 · answered Mar 28 '22 at 18:24

This is the Terraform version of using the --from-file flag with kubectl.

Basically, you'll add the name of the file and its contents per the structure of the data block below.

resource "kubernetes_secret" "azure_config_file" {
  metadata {
    name = "azure-config-file"
  }

  data = { "azure.json" = jsonencode({
    tenantId        = data.azurerm_subscription.current.tenant_id
    subscriptionId  = data.azurerm_subscription.current.subscription_id
    resourceGroup   = data.azurerm_resource_group.rg.name
    aadClientId     = azuread_application.sp_externaldns_connect_to_dns_zone.application_id
    aadClientSecret = azuread_application_password.sp_externaldns_connect_to_dns_zone.value
    })

  }
}

How can I automatically add a kubernetes client secret as a file mount in Terraform?

1 Answers1