0

We created the GKE cluster with the public endpoint. The service account of the GKE cluster and Node pool has the following roles.

"roles/compute.admin",
"roles/compute.viewer",
"roles/compute.securityAdmin",
"roles/iam.serviceAccountUser",
"roles/iam.serviceAccountAdmin",
"roles/resourcemanager.projectIamAdmin",
"roles/container.admin",
"roles/artifactregistry.admin",
"roles/storage.admin"

The node pool of the GKE cluster has the following OAuth scopes

"https://www.googleapis.com/auth/cloud-platform",
"https://www.googleapis.com/auth/devstorage.read_write",

The private GCS bucket has the same service account as the principal, with the storage admin role.

When we try to read/write in this bucket from a GKE POD, we get the below error.

# Read
AccessDeniedException: 403 Caller does not have storage.objects.list access to the Google Cloud Storage bucket

# Write
AccessDeniedException: 403 Caller does not have storage.objects.create access to the Google Cloud Storage object

We also checked this thread but the solution was credential oriented and couldn't help us. We would like to read/write without maintaining the SA auth key or any sort of credentials.

Please guide what is missing here.

UPDATE: as per the suggestion by @boredabdel we checked and found that workload identity was already enabled on the GKE cluster as well as NodePool. We are using this module to create our cluster where it is already enabled by default. Still, we are facing connectivity issues.

Cluster Security:

enter image description here

NodePool Security:

enter image description here

Nitin G
  • 714
  • 7
  • 31

2 Answers2

1

Seems like you are trying to use the Node Service Account to authenticate to GCS. You need to passe the Service Account Key to the app you are calling the API from as described in this doc

If you want Keyless authentication, my advice is to use Workload identity

boredabdel
  • 1,732
  • 3
  • 7
  • Hi, @boredabdel thank you for your suggestion. It looks like WI is already enabled by default in our cluster. I have updated the question with the relevant info. Please check and suggest. – Nitin G Aug 10 '22 at 07:05
  • Enabled is not enough. You have to configure both your workload and IAM for Workload Identity to work. You can check this documentation https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to – boredabdel Aug 10 '22 at 07:20
  • Hey, @boredabdel thanks for your help. We were able to configure the workload identity and access GCS without keys. – Nitin G Aug 12 '22 at 09:53
1

This is for all who are looking for an answer to implement this solution via terraform. Please refer below:

Create a Kubernetes service account with annotation
resource "kubernetes_manifest" "service_account" {
  manifest = {
    "apiVersion" = "v1"
    "kind"       = "ServiceAccount"
    "metadata" = {
      "name"      = "KSA_NAME"
      "namespace" = "NAME"
      "annotations" = {
        "iam.gke.io/gcp-service-account" = "GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com"
      }
    }
    "automountServiceAccountToken" = true
  }
}
Create an IAM policy binding to allow the Kubernetes service account to act as the IAM service account.
resource "google_service_account_iam_binding" "service-account-iam" {
  service_account_id = "projects/PROJECT_ID/serviceAccounts/GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com"
  role               = "roles/iam.workloadIdentityUser"
  members            = [
    "serviceAccount:${var.project_id}.svc.id.goog[NAMESPACE/KSA-NAME]",
  ]
}
Add service account to the deployment/statefulset manifest like this
spec:
  serviceAccount: KSA-NAME
  containers:

NOTE:

  1. GSA_PROJECT and PROJECT_ID are the same if you are using the same project for all objects.
  2. K8s service account is created via resource "kubernetes_manifest" method because there is an open issue for it to create via resource "kubernetes_service_account" method where it is throwing an error stating. Waiting for default secret of "NAMESPACE/KSA_NAME" to appear
Nitin G
  • 714
  • 7
  • 31