0

I am trying to use Google's preferred "Workload Identity" method to enable my GKE app to securely access secrets from Google Secrets.

I've completed the setup and even checked all steps in the Troubleshooting section (https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity?hl=sr-ba#troubleshooting) but I'm still getting the following error in my logs:

Unhandled exception. Grpc.Core.RpcException: Status(StatusCode=PermissionDenied, Detail="Permission 'secretmanager.secrets.list' denied for resource 'projects/my-project' (or it may not exist).")

I figured the problem was due to the node pool not using the correct service account, so I recreated it, this time specifying the correct service account.

The service account has the following roles added:

  • Cloud Build Service
  • Account Kubernetes Engine Developer
  • Container Registry Service Agent
  • Secret Manager Secret Accessor
  • Secret Manager Viewer

The relevant source code for the package I am using to authenticate is as follows:

var data = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);

var request = new ListSecretsRequest
{
    ParentAsProjectName = ProjectName.FromProject(projectName),
};

var secrets = secretManagerServiceClient.ListSecrets(request);
foreach(var secret in secrets)
{
    var value = secretManagerServiceClient.AccessSecretVersion($"{secret.Name}/versions/latest");
    string secretVal = this.manager.Load(value.Payload);
    string configKey = this.manager.GetKey(secret.SecretName);
    data.Add(configKey, secretVal);
}
Data = data;

Ref. https://github.com/jsukhabut/googledotnet

Am I missing a step in the process?

Any idea why Google is still saying "Permission 'secretmanager.secrets.list' denied for resource 'projects/my-project' (or it may not exist)?"

user1477388
  • 20,790
  • 32
  • 144
  • 264
  • 2
    How have you mapped that service account to your pod? Workload Identity _does not_ use the underlying node identity and instead maps a k8s service account to a gcp service account. – sethvargo Jun 16 '21 at 19:04
  • I don't think I've mapped the k8s service account to the pod; I only created it and annotated is per the GKE workload identity documentation. You're saying I need to map the k8s service account to the pod which will be intercepted and make use of my annotated Google service account? If so, how can I map my k8s service account to my pod or should I map it to the node pool so new pods will include the mapping, too? – user1477388 Jun 16 '21 at 19:35
  • @sethvargo I found this https://cloud.google.com/kubernetes-engine/docs/how-to/kubernetes-service-accounts#assigning_a_kubernetes_service_account_to_a_pod but I'm not sure how to do this at the node pool level so all future pods will receive this service account config. Any idea? Maybe I am misunderstanding how it works. – user1477388 Jun 16 '21 at 19:52
  • The point of workload identity is that you no longer do anything at the node pool level. Everything happens at the per-pod level. You create a GCP service account with the required permissions. You create a K8S service account. You give the K8S service account permission to impersonate the GCP service account. You run your workload as the K8S service account. – sethvargo Jun 16 '21 at 19:58
  • @sethvargo Right, but doesn't the node pool create the pods - if I create a pod manually using a YAML file, what will happen if the pod is destroyed or upgraded? Sorry for the fundamental questions - I am not sure how manual pod creation works within the context of GKE. Is it best practice to add the YAML file to source control somewhere also, do you know? Thanks for any advice! – user1477388 Jun 16 '21 at 20:15
  • There are some default ("system") pods, but no, most pods are created by you or a CI/CD tool. There are *many* ways to manage K8S manifests (raw, helm, kept, ...). – sethvargo Jun 16 '21 at 20:19
  • @sethvargo My cluster is set up for CI/CD, so do you know if there's a way to edit my deployment.yaml file to make the recommended change of specifying `serviceAccountName` for the pod? Apparently, Google Cloud Build is creating pods based on the deployment.yaml file, but I can't find any docs on how to edit it to add the `serviceAccountName`. – user1477388 Jun 16 '21 at 20:42
  • @sethvargo I've tried adding it to the deployment.yaml file under spec -> containers, but the build fails with `unknown field "serviceAccountName" in io.k8s.api.core.v1.Container`. Any idea how I can add the service account for my pods? Thanks for your help! – user1477388 Jun 16 '21 at 21:43
  • https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/ – sethvargo Jun 17 '21 at 14:43
  • @sethvargo Thanks, I've seen that article but my skill level with k8s is insufficient for understanding how to apply it within the context of a GKE CI/CD config. I have asked another question at https://stackoverflow.com/questions/68021355/how-to-specify-serviceaccountname-for-pods-in-gcp-deployment-yaml to help clear up my confusion. Feel free to post your initial suggestion as an answer to this question: "Workload Identity does not use the underlying node identity and instead maps a k8s service account to a gcp service account. You run your workload as the K8S service account." – user1477388 Jun 17 '21 at 14:45

1 Answers1

3

Like @sethvargo mentioned in the comments, you need to map the service account to your pod because Workload Identity doesn’t use the underlying node identity and instead maps a Kubernetes service account to a GCP service account. Everything happens at the per-pod level in Workload identity.

Assign a Kubernetes service account to the application and configure it to act as a Google service account.

1.Create a GCP service account with the required permissions.

2.Create a Kubernetes service account.

3.Assign the Kubernetes service account permission to impersonate the GCP service account.

4.Run your workload as the Kubernetes service account.

Hope you are using project ID instead of project name in the project or secret.

You cannot update the service account of an already created pod.

Refer the link to add service account to the pods.

Jyothi Kiranmayi
  • 2,090
  • 5
  • 14