Unable to run 'kubectl' commands after using impersonation to fetch GKE cluster credentials

Question

My Objective

I want to use GCP impersonation to fetch my GKE cluster credentials. And then I want to run kubectl commands.

Initial Context

1. I have a GCP project called rakib-example-project
2. I have 2 ServiceAccounts in the project called:
   • owner@rakib-example-project.iam.gserviceaccount.com
     • it has the project-wide roles/owner role - so it can do anything and everything inside the GCP project
   • executor@rakib-example-project.iam.gserviceaccount.com
     • it only has the project-wide roles/iam.serviceAccountTokenCreator role - so it can impersonate the owner ServiceAccount in the GCP project
3. I have 1 GKE cluster in the project called my-gke-cluster

The Problem

✅ I have authenticated as the executor ServiceAccount:

$ gcloud auth activate-service-account --key-file=my_executor_sa_key.json

Activated service account credentials for: [executor@rakib-example-project.iam.gserviceaccount.com]

✅ I have fetched GKE cluster credentials by impersonating the owner:

$ gcloud container clusters get-credentials my-gke-cluster \
  --zone asia-southeast1-a \
  --project rakib-example-project \
  --impersonate-service-account=owner@rakib-example-project.iam.gserviceaccount.com

WARNING: This command is using service account impersonation. All API calls will be executed as [owner@rakib-example-project.iam.gserviceaccount.com].
WARNING: This command is using service account impersonation. All API calls will be executed as [owner@rakib-example-project.iam.gserviceaccount.com].
Fetching cluster endpoint and auth data.
kubeconfig entry generated for my-gke-cluster.

❌ I am failing to list cluster nodes due to missing container.nodes.list permission:

$ kubectl get nodes

Error from server (Forbidden): nodes is forbidden: User "executor@rakib-example-project.iam.gserviceaccount.com" cannot list resource "nodes" in API group "" at the cluster scope: requires one of ["container.nodes.list"] permission(s).

But I have already impersonated the Owner ServiceAccount. Why would it still have missing permissions?

My Limitations

It works well if i grant my executor ServiceAccount the roles/container.admin role. However, I am not allowed to grant such roles to my executor ServiceAccount due to compliance requirements. I can only impersonate the owner ServiceAccount and THEN do whatever I want through it - not directly.

Answer 1

If you have a look to your kubeconfig file at this location ~/.kube/config, you can see the list of authorization and the secrets, such as

- name: gke_gdglyon-cloudrun_us-central1-c_knative
  user:
    auth-provider:
      config:
        access-token: ya29.<secret>-9XQmaTQodj4kS39w
        cmd-args: config config-helper --format=json
        cmd-path: /usr/lib/google-cloud-sdk/bin/gcloud
        expiry: "2020-08-25T17:48:39Z"
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp

You see external references (expiry-key and token-key) and a cmd-path. The command path is interesting because when a new token need to be generated, it will be called.

However, you see any mention of the impersonation. You have to add it in the command path, to be used by default. For this, add it in your config like this:

gcloud config set auth/impersonate_service_account owner@rakib-example-project.iam.gserviceaccount.com

Now, every use of the gcloud CLI will use the impersonate service account, and it's what you want to generate a valid access_token to reach your GKE cluster