Authenticating on AKS for deploying a Helm release with Terraform

Question

I am trying to do a Helm chart deployment through Terraform code on AKS. The TF code that I have will create a resource in Datadog from which I will grab an output value that will be passed to my Helm release to be deployed on my cluster. It only has to create two resources, one of which is the Helm chart.
The problem that I am having is with authentication against my Kubernetes cluster, I am using a data source to bring the credentials from the cluster and then pass them in my kubernetes and helm providers.

My Terraform state for the AKS cluster is stored inside a Blob in a Azure Storage account.

I have tried updating the Helm chart versions, using different methods to access the data such as ${} around my variables. Tried changing from username = data.azurerm_kubernetes_cluster.credentials.kube_config.0.username to use the admin configuration username = data.azurerm_kubernetes_cluster.credentials.kube_admin_config.0.username Tried

Terraform version: 1.1.7

A data source is setup to bring the credentials for the AKS cluster in main.tf

data "azurerm_kubernetes_cluster" "credentials" {
  name                = var.aks_cluster_name
  resource_group_name = var.aks_cluster_resource_group_name
}

This is versions.tf and what is being used to setup the connections to AKS.

terraform {
  required_providers {
    datadog = {
      source = "DataDog/datadog"
    }
  }
  backend "azurerm" {
  }
}

provider "azurerm" {
  features {}
}

provider "helm" {
  debug = true
  kubernetes {
    username               = data.azurerm_kubernetes_cluster.credentials.kube_config.0.username
    password               = data.azurerm_kubernetes_cluster.credentials.kube_config.0.password
    host                   = data.azurerm_kubernetes_cluster.credentials.kube_config.0.host
    client_certificate     = base64decode(data.azurerm_kubernetes_cluster.credentials.kube_config.0.client_certificate)
    client_key             = base64decode(data.azurerm_kubernetes_cluster.credentials.kube_config.0.client_key)
    cluster_ca_certificate = base64decode(data.azurerm_kubernetes_cluster.credentials.kube_config.0.cluster_ca_certificate)
  }
}

provider "kubernetes" {
  username               = data.azurerm_kubernetes_cluster.credentials.kube_config.0.username
  password               = data.azurerm_kubernetes_cluster.credentials.kube_config.0.password
  host                   = data.azurerm_kubernetes_cluster.credentials.kube_config.0.host
  client_certificate     = base64decode(data.azurerm_kubernetes_cluster.credentials.kube_config.0.client_certificate)
  client_key             = base64decode(data.azurerm_kubernetes_cluster.credentials.kube_config.0.client_key)
  cluster_ca_certificate = base64decode(data.azurerm_kubernetes_cluster.credentials.kube_config.0.cluster_ca_certificate)
}

Error that I am seeing when running terraform apply, which will report that it can't find the elements in the collection for any of the attributes specified in my provider:

╷
│ Error: Invalid index
│ 
│   on versions.tf line 26, in provider "helm":
│   26:     host                   = data.azurerm_kubernetes_cluster.credentials.kube_admin_config.0.host
│     ├────────────────
│     │ data.azurerm_kubernetes_cluster.credentials.kube_admin_config has a sensitive value
│ 
│ The given key does not identify an element in this collection value.
╵
[ ... ]
╷
│ Error: Invalid index
│ 
│   on versions.tf line 27, in provider "helm":
│   27:     username               = data.azurerm_kubernetes_cluster.credentials.kube_admin_config.0.username
│     ├────────────────
│     │ data.azurerm_kubernetes_cluster.credentials.kube_admin_config has a sensitive value
│ 
│ The given key does not identify an element in this collection value.

I am unsure on how to change my Terraform code such that this authentication works, given that the methods mentioned above have yielded no results. If needed I can provide the TF code for the deployment of the resources.

@MarkoE Yes, I've added in the post how my data source is defined, but I'll add it in the comments again `data "azurerm_kubernetes_cluster" "credentials" {}` — Julanu, Sep 26 '22 at 10:21
Sorry, my bad. Can you try `nonsensitive(data.azurerm_kubernetes_cluster.credentials.kube_config.0.username)` and `nonsensitive(data.azurerm_kubernetes_cluster.credentials.kube_config.0.host)`? — Marko E, Sep 26 '22 at 10:25
@MarkoE after trying your suggestion, I am still getting the same error of "..given key does not identify an element in this collection value." — Julanu, Sep 26 '22 at 12:00
Ah, it should have been `nonsensitive(data.azurerm_kubernetes_cluster.credentialskube_admin_config.0.username)` and `nonsensitive(data.azurerm_kubernetes_cluster.credentialskube_admin_config.0.host)`. — Marko E, Sep 26 '22 at 12:46
@MarkoE Doesn't seem to be working, I get the same error but if I am using `kube_config` instead of `kube_admin_config` with `nonsensitive` I get the following error ` Error: Kubernetes cluster unreachable: the server has asked for the client to provide credentials` which I am digging into now — Julanu, Sep 26 '22 at 13:04

Daniel Argüelles · Answer 1 · 2022-11-25T10:15:23.060

I'm using kubelogin to identify myself:


data "azurerm_client_config" "current" {
}


provider "helm" {
  kubernetes {
    host = azurerm_kubernetes_cluster.aks.kube_config.0.host
    cluster_ca_certificate = base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.cluster_ca_certificate)

    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      args = [
        "get-token",
        "--environment", "AzurePublicCloud",
        "--server-id", "6dae42f8-4368-4678-94ff-3960e28e3630", # The AAD server app ID of AKS Managed AAD is always 6dae42f8-4368-4678-94ff-3960e28e3630 in any environments.
        "--client-id", "${yamldecode(azurerm_kubernetes_cluster.aks.kube_config_raw).users[0].user.auth-provider.config.client-id}", 
        "--tenant-id", data.azurerm_client_config.current.tenant_id,
        "--login", "devicecode"
      ]
      command = "kubelogin"
    }
  }
}

Authenticating on AKS for deploying a Helm release with Terraform

1 Answers1