GCP SQL proxy Unable to generate access token; IAM returned 403 Forbidden: The caller does not have permission

Question

I'm having a k8s deployment on GCP, and I'm connecting to the SQL database with SQL proxy. I'm deploying with a service account. I gave the service account owner permissions (it's just a test project), and CloudSQL client, CloudSQL instance user, CloudSQL admin.

The whole deployment goes all good, but it cannot connect to the proxy saying:

errors parsing config:
    Get "https://sqladmin.googleapis.com/sql/v1beta4/projects/org/instances/europe-west2~testProject/connectSettings?alt=json&prettyPrint=false": compute: Received 403 `Unable to generate access token; IAM returned 403 Forbidden: The caller does not have permission
This error could be caused by a missing IAM policy binding on the target IAM service account.
For more information, refer to the Workload Identity documentation:
    https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to

And this is the manifest file (relevant parts of it):

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    iam.gke.io/gcp-service-account: testProject@org.iam.gserviceaccount.com
  name: testProject
  labels:
    app.kubernetes.io/instance: testProject
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: testProject
    app.kubernetes.io/instance: testProject
  name: testProject
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: testProject
  strategy:
    rollingUpdate:
      maxSurge: 2
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: testProject
    spec:
      serviceAccountName: testProject
      containers:
        # app
        - name: cloud-sql-proxy
          command:
            - /cloud_sql_proxy
            - -enable_iam_login
            - -instances=org:europe-west2:testProject-db-staging=tcp:5432
            - -ip_address_types=PRIVATE,PUBLIC
            - -structured_logs
            - -term_timeout=5s
          image: gcr.io/cloudsql-docker/gce-proxy:1.30.0
          # ...options

Why can't it connect to the database?

What's the permission of the testProject service account binding? — guillaume blaquiere, Nov 23 '22 at 21:06
Did you complete step 3 to grant permission to use the service account: https://cloud.google.com/config-connector/docs/how-to/install-upgrade-uninstall#identity — John Hanley, Nov 23 '22 at 21:55
Isn't the binding/mapping to k8s service account done automatically via the annotation? — Gergő Horváth, Nov 24 '22 at 10:37
@guillaumeblaquiere ` roles/cloudsql.admin roles/cloudsql.client roles/cloudsql.instanceUser roles/owner roles/resourcemanager.projectIamAdmin` — Gergő Horváth, Nov 24 '22 at 19:39

score 0 · Answer 1 · answered Jan 20 '23 at 13:48

Use the answer described here: How to Connect to Cloud SQL Through Kubernetes

The key is to the follow the tutorial very carefully. Few things to remember that I didn't realize I was missing:

svc.id.goog service account naming convention is very specific. Be sure you write it properly when adding it to the workloadIdentityUser group
make sure your service account is configured with the necessary roles (i.e. a member of the proper IAM roles). I think the key ones to make things work are cloudsql, iam.ServiceAccount..., and iam.workloadIdentityUser (e.g. iam.serviceAccountClient or other ones)
Make sure the k8 cluster worker pool is updated to use the Workload Identity you create, either by associating the pool during creation time, or by running gcloud container node-pools update POOL_NAME. If you didnt create a pool the POOL_NAME will be your default pool

GCP SQL proxy Unable to generate access token; IAM returned 403 Forbidden: The caller does not have permission

1 Answers1