3

I can use terraform to deploy a Kubernetes cluster in GKE.

Then I have set up the provider for Kubernetes as follows:

provider "kubernetes" {
  host                    = "${data.google_container_cluster.primary.endpoint}"

  client_certificate      = "${base64decode(data.google_container_cluster.primary.master_auth.0.client_certificate)}"
  client_key              = "${base64decode(data.google_container_cluster.primary.master_auth.0.client_key)}"
  cluster_ca_certificate  = "${base64decode(data.google_container_cluster.primary.master_auth.0.cluster_ca_certificate)}"
}

By default, terraform interacts with Kubernetes with the user client, which has no power to create (for example) deployments. So I get this error when I try to apply my changes with terraform:

Error: Error applying plan:

1 error(s) occurred:

 * kubernetes_deployment.foo: 1 error(s) occurred:

 * kubernetes_deployment.foo: Failed to create deployment: deployments.apps is forbidden: User "client" cannot create deployments.apps in the namespace "default"

I don't know how should I proceed now, how should I give this permissions to the client user?

If the following fields are added to the provider, I am able to perform deployments, although after reading the documentation it seems these credentials are used for HTTP communication with the cluster, which is insecure if it is done through the internet.

username              = "${data.google_container_cluster.primary.master_auth.0.username}"
password              = "${data.google_container_cluster.primary.master_auth.0.password}"

Is there any other better way of doing so?

Navarro
  • 1,284
  • 2
  • 17
  • 40
  • If you find additional ressources on how to do this, feel free to edit your question. I'm currently stuck on the exact same problem, there does not seem to be a complete end-to-end exemple on how to deploy a GKE Cluster and a Kubernetes deployment inside it with the newer RBAC. – b4stien Mar 03 '19 at 13:03

3 Answers3

9
  • you can use the service account that are running the terraform 
data "google_client_config" "default" {}

provider "kubernetes" {
  host     = "${google_container_cluster.default.endpoint}"

  token = "${data.google_client_config.default.access_token}"
  cluster_ca_certificate = "${base64decode(google_container_cluster.default.master_auth.0.cluster_ca_certificate)}"

  load_config_file = false
}

OR

  • give permissions to the default "client"
  • But you need a valid authentication on GKE cluster provider to run this :/ ups circular dependency here 
resource "kubernetes_cluster_role_binding" "default" {
  metadata {
    name = "client-certificate-cluster-admin"
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind = "ClusterRole"
    name = "cluster-admin"
  }
  subject {
    kind = "User"
    name = "client"
    api_group = "rbac.authorization.k8s.io"
  }
  subject {
    kind = "ServiceAccount"
    name = "default"
    namespace = "kube-system"
  }
  subject {
    kind = "Group"
    name = "system:masters"
    api_group = "rbac.authorization.k8s.io"
  }
}
Abel Matos
  • 171
  • 6
  • Thank you! My terraform was running with a GCP identity that had the necessary rights to the cluster, but I couldn't figure out how to configure the Kubernetes provider to use these credentials. Your first example, showing how to use "google_client_config" to get the OAuth token was what I needed. If anybody is in a similar situation, I'll add that you have to make sure you aren't specifying a "client_certificate" or "client_key" value when using this solution -- only "token" should be supplied. – Sean Kleinjung Nov 29 '19 at 06:23
0

It looks like the user that you are using is missing the required RBAC role for creating deployments. Make sure that user has the correct verbs for the deployments resource. You can take a look at this Role examples to have an idea about it.

ozrlz
  • 84
  • 2
0

You need to provide both. Check this example on how to integrate the Kubernetes provider with the Google Provider.

Example of how to configure the Kubernetes provider:

provider "kubernetes" {
  host     = "${var.host}"
  username = "${var.username}"
  password = "${var.password}"

  client_certificate     = "${base64decode(var.client_certificate)}"
  client_key             = "${base64decode(var.client_key)}"
  cluster_ca_certificate = "${base64decode(var.cluster_ca_certificate)}"
}
Eduardo Ruela
  • 761
  • 4
  • 8