0

Intro

I'm having an issue with the Kubernetes provider in Terraform. When I configure it with my local ~/.kube/config, it works. But when I try to configure it with the outputs from my cluster creation module, it mysteriously tries to act as a user named 'client' who has no permissions within the cluster, and terraform plan fails.

My code (outputs from cluster creation)

# module 'cluster'

output "cluster_endpoint" {
  value = google_container_cluster.general_purpose.endpoint
}

output "cluster_client_certificate" {
  value = base64decode(google_container_cluster.general_purpose.master_auth[0].client_certificate)
}

output "cluster_client_key" {
  value     = base64decode(google_container_cluster.general_purpose.master_auth[0].client_key)
  sensitive = true
}

output "cluster_ca_certificate" {
  value = base64decode(google_container_cluster.general_purpose.master_auth[0].cluster_ca_certificate)
}

What works

# module 'ingress'

provider "kubernetes" {
  config_path = "~/.kube/config"
}

data "kubernetes_namespace" "namespace_default" {
  metadata {
    name = "default"
  }
}

What doesn't work

# module 'ingress'

provider "kubernetes" {
  host                   = "https://${var.cluster_endpoint}"
  client_certificate     = var.cluster_client_certificate
  client_key             = var.cluster_client_key
  cluster_ca_certificate = var.cluster_ca_certificate
}

data "kubernetes_namespace" "namespace_default" {
  metadata {
    name = "default"
  }
}

The Error

$ terraform plan

╷
│ Error: namespaces "default" is forbidden: User "client" cannot get resource "namespaces" in API group "" in the namespace "default"
│ 
│   with module.ingress.data.kubernetes_namespace.namespace_default,
│   on modules/ingress/main.tf line 42, in data "kubernetes_namespace" "namespace_default":
│   42: data "kubernetes_namespace" "namespace_default" {
│ 
╵

I haven't been able to figure out where this user named 'client' is set. I can replicate the error by running the command using the --as client argument in kubectl. (The same command works if I omit this argument.)

$ kubectl get namespace default --as client

Error from server (Forbidden): namespaces "default" is forbidden: User "client" cannot get resource "namespaces" in API group "" in the namespace "default"

I'm under the impression that there should be no "named" user involved, because I am working with certificate-based authentication. But somehow, a username is working its way into the command used by the Kubernetes provider.

I appreciate any suggestions!

Cameron Hudson
  • 3,190
  • 1
  • 26
  • 38
  • Is it possible that your client-key is not for the client user, but someone else? Also if it is possible I recommend using the actual kube config, I am using the rke provider and output it's kube_config_cluster.yml which I use as the helm provider's config_path and haven't run into any issues so far, but it's the only approach I could get working. – jfh Sep 21 '21 at 02:16
  • I used my local kubeconfig with success. However, I don't want the helm provider to refer to my local state. Instead, I'm directly referring to outputs from a `google_container_cluster` terraform resource. (I've add the origins of my `var`s to my post.) Interestingly, the user `aaaaahaaaaa` has almost exactly the same issue as me in this thread: https://github.com/hashicorp/terraform-provider-helm/issues/647#issuecomment-756799242. – Cameron Hudson Sep 22 '21 at 16:12
  • 1
    @jfh I have an update: My issue is not specific to the `helm` provider. The `kubernetes` provider behaves the same way (I reckon that this is because the `helm` provider relies on the `kubernetes` provider.) I've updated my question with a more generic example. I'll do more research from the `kubernetes` provider side, as well. – Cameron Hudson Sep 26 '21 at 14:32
  • 1
    @jfh I found that it was a bug in GKE. GCP creates the master authentication `client_certificate` with a username of `client`. However. No `clusterrolebinding` is created for that user. The workaround was to use my `google_client_config` to create a `clusterrolebinding` between user `client` and the `clusterrole` `cluster-admin` when the cluster was created. (Refer to dupe question link.) Thereafter, cert-based authentication worked as expected. – Cameron Hudson Sep 26 '21 at 16:11

0 Answers0