Failed to retrieve token from the Google Compute Engine metadata service. Status: 404

Question

I'm trying to set up Cloud SQL Proxy running as a sidecar in my GKE cluster. The configuration is done via Terraform. I've set up workload identity, required service accounts, and so on. When launching ./cloud_sql_proxy from within the GKE cluster (kubectl run -it --image google/cloud-sdk:slim --serviceaccount ksa-name --namespace k8s-namespace workload-identity-test), I get the following output:

root@workload-identity-test:/# ./cloud_sql_proxy -instances=project-id:europe-west4:db-instance=tcp:5432
2020/11/24 17:18:39 current FDs rlimit set to 1048576, wanted limit is 8500. Nothing to do here.
2020/11/24 17:18:40 GcloudConfig: error reading config: exit status 1; stderr was:
ERROR: (gcloud.config.config-helper) There was a problem refreshing your current auth tokens: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/db-proxy@project-id.iam.gserviceaccount.com/token from the Google Compute Enginemetadata service. Status: 404 Response:\nb'Unable to generate access token; IAM returned 404 Not Found: Requested entity was not found.\\n'", <google_auth_httplib2._Response object at 0x7fc5575545f8>)
Please run:

  $ gcloud auth login

to obtain new credentials.

If you have already logged in with a different account:

    $ gcloud config set account ACCOUNT

to select an already authenticated account to use.
2020/11/24 17:18:41 GcloudConfig: error reading config: exit status 1; stderr was:
ERROR: (gcloud.config.config-helper) There was a problem refreshing your current auth tokens: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/db-proxy@project-id.iam.gserviceaccount.com/token from the Google Compute Enginemetadata service. Status: 404 Response:\nb'Unable to generate access token; IAM returned 404 Not Found: Requested entity was not found.\\n'", <google_auth_httplib2._Response object at 0x7f06f72f45c0>)
Please run:

  $ gcloud auth login

to obtain new credentials.

If you have already logged in with a different account:

    $ gcloud config set account ACCOUNT

to select an already authenticated account to use.
2020/11/24 17:18:41 errors parsing config:
        Get "https://sqladmin.googleapis.com/sql/v1beta4/projects/project-id/instances/europe-west4~db-instance?alt=json&prettyPrint=false": metadata: GCE metadata "instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fsqlservice.admin" not defined

Here's troubleshooting I've done so far:

root@workload-identity-test:/# gcloud auth list
                   Credentialed Accounts
ACTIVE  ACCOUNT
*       db-proxy@project-id.iam.gserviceaccount.com

To set the active account, run:
    $ gcloud config set account `ACCOUNT`

λ gcloud container clusters describe mycluster --format="value(workloadIdentityConfig.workloadPool)"
project-id.svc.id.goog

λ gcloud container node-pools describe mycluster-node-pool --cluster=mycluster --format="value(config.workloadMetadataConfig.mode)"
GKE_METADATA

λ gcloud container node-pools describe mycluster-node-pool --cluster=mycluster--format="value(config.oauthScopes)"
https://www.googleapis.com/auth/monitoring;https://www.googleapis.com/auth/devstorage.read_only;https://www.googleapis.com/auth/logging.write;https://www.googleapis.com/auth/cloud-platform;https://www.googleapis.com/auth/userinfo.email;https://www.googleapis.com/auth/compute;https://www.googleapis.com/auth/sqlservice.admin

λ kubectl describe serviceaccount --namespace k8s-namespace ksa-name
Name:                ksa-name
Namespace:           k8s-namespace
Labels:              <none>
Annotations:         iam.gke.io/gcp-service-account: db-proxy@project-id.iam.gserviceaccount.com
Image pull secrets:  <none>
Mountable secrets:   ksa-name-token-87n4t
Tokens:              ksa-name-token-87n4t
Events:              <none>

λ gcloud iam service-accounts get-iam-policy db-proxy@project-id.iam.gserviceaccount.com
bindings:
- members:
  - serviceAccount:project-id.svc.id.goog[k8s-namespace/ksa-name]
  role: roles/iam.workloadIdentityUser
etag: BwW02zludbY=
version: 1

λ kubectl get networkpolicy --namespace k8s-namespace
No resources found in k8s-namespace namespace.

λ gcloud projects get-iam-policy project-id
bindings:
- members:
  - serviceAccount:db-proxy@project-id.iam.gserviceaccount.com
  role: roles/cloudsql.editor

Expected result (I got this running on another cluster and changed configuration afterwards, can't find where my mistake is):

root@workload-identity-test:~# ./cloud_sql_proxy -instances=project-id:europe-west4:db-instance-2=tcp:5432
2020/11/24 18:09:54 current FDs rlimit set to 1048576, wanted limit is 8500. Nothing to do here.
2020/11/24 18:09:56 Listening on 127.0.0.1:5432 for project-id:europe-west4:db-instance-2
2020/11/24 18:09:56 Ready for new connections

What am I doing wrong? How do I troubleshoot or debug further?

Answer 1

This could be due to the service account not being enabled when the Kubernetes cluster was created, or it wasn't configured properly. Try checking if the Service Account is disabled and Enable if it is. You could also try to create a new service account and change the service account in the pods. Or finally, try to provide the credentials to the gcloud command when running.

Answer 2

I was able to resolve the problem by creating a service account with a different name. Just the name has changed, nothing else. If I delete the db-proxy@project-id.iam.gserviceaccount.com and then use the name again, the problem still persists. I was not able to find any other reference to said account. The problem was not encountered again after my comment on Nov 30 '20.

Answer 3

Could you confirm that 'db-proxy@project-id.iam.gserviceaccount.com' is the correct account? I may be reading it wrong, but it seems that there is an error trying to refresh the auth token for that account, and the error is that the account does not exist.

Answer 4

I encountered a similar error today and discovered that it was because the GSA was in a different project from the GKE cluster. It seem like the iam.workloadIdentityUser binding needs to be between accounts in the same project.

So this worked:

gcloud iam service-accounts create custom-metrics-adapter \
    --project ${PLATFORM_PROJECT_ID}

gcloud iam service-accounts add-iam-policy-binding \
    "${GSA_NAME}@${PLATFORM_PROJECT_ID}.iam.gserviceaccount.com" \
    --member "serviceAccount:${PLATFORM_PROJECT_ID}.svc.id.goog[${KSA_NAMESPACE}/${KSA_NAME}]" \
    --role "roles/iam.workloadIdentityUser" \
    --project ${PLATFORM_PROJECT_ID}

with

apiVersion: v1
kind: ServiceAccount
metadata:
  name: ${KSA_NAME}
  namespace: ${KSA_NAMESPACE}
  annotations:
    iam.gke.io/gcp-service-account: ${GSA_NAME}${PLATFORM_PROJECT_ID}.iam.gserviceaccount.com
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: example
  namespace: ${KSA_NAMESPACE}
spec:
  template:
    spec:
      serviceAccountName: ${KSA_NAME}
# Deployment spec truncated for clarity

Not sure if that helps you, but maybe it will help someone else who finds this by searching the error string:

Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/${GSA_NAME}@${DIFFERENT_PROJECT_ID}.iam.gserviceaccount.com/token from the Google Compute Enginemetadata service. Status: 404 Response:\nb'Unable to generate access token; IAM returned 404 Not Found: Requested entity was not found.

Answer 5

In my case I wasn't able to rename the service account due to other automation abstractions, but I was able to resolve this issue by deleting the Kubernetes namespace and then recreating it from scratch.

kubectl delete namespace <your namespace>

Just note that this deletes everything in your Kubernetes namespace, so don't do this for any system that's in use.