Accessing Sheets API from GKE using ServiceAccount

Question

My attempts to reach Google Sheets API from within GKE fail with a 403 HTTP code.

This is the first time I am trying to reach non-Google Cloud APIs, e.g. Google Sheets API, using a service account. I think I may be categorically wrong in a way I am doing it.

Could someone help review my thinking process below to see where I might be making a wrong turn?

I've a Pod running with a K8s services account that:

• is impersonating a Google IAM ServiceAccount through Workload Identity mechanism,

• for said Google IAM ServiceAccount, its ClientID + a single scope https://www.googleapis.com/auth/spreadsheets.readonly are used to enable domain-wide delegation at https://admin.google.com/ac/owl/domainwidedelegation,

Here's what I am doing as a test:

1. ssh into pod,

2. fetch the access_token from Metadata server:

   curl \
     --silent \
     --header 'metadata-flavor: Google' \
     http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token | jq '.access_token'
   
   # removed the weird trailing dots after acces_token value...
   

3. try to make an HTTP request to Google Spreadsheets API:

   curl \
     --silent \
     --header "Authorization: Bearer $access_token" \
     https://sheets.googleapis.com/v4/spreadsheets/28PGMGj1sc1ahn2Qzr6bpCbRHESSZp9DgbCLZsloVxvf
   

4. Get a response:

   {
     "error": {
       "code": 403,
       "message": "Request had insufficient authentication scopes.",
       "status": "PERMISSION_DENIED",
       "details": [
         {
           "@type": "type.googleapis.com/google.rpc.ErrorInfo",
           "reason": "ACCESS_TOKEN_SCOPE_INSUFFICIENT",
           "domain": "googleapis.com",
           "metadata": {
             "method": "google.apps.sheets.v4.SpreadsheetsService.GetSpreadsheet",
             "service": "sheets.googleapis.com"
           }
         }
       ]
     }
   }
   

Checking the scopes available for the token indeed do not reveal spreadsheets to be present:

• request:

   curl --header 'Metadata-Flavor: Google' \ 
     http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/scopes
  

• response:

   https://www.googleapis.com/auth/cloud-platform
  

Inside GKE, how can I generate a ServiceAccount token that has sufficient privileges to access Sheets API?

In another attempt based on this SO answer, I included scopes in the GET request to metadata server. Interestingly, this time I get a 403 of a different JSON shape. Steps:

1. request the token, include a scopes=https://www.googleapis.com/auth/spreadsheets.readonl GET parameter:

   curl \
     --silent \
     --header 'metadata-flavor: Google' \
     http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token?scopes=https://www.googleapis.com/auth/spreadsheets.readonly
   

2. try to make an HTTP request to Google Spreadsheets API:

   curl \
     --silent \
     --header "Authorization: Bearer $access_token" \
     https://sheets.googleapis.com/v4/spreadsheets/28PGMGj1sc1ahn2Qzr6bpCbRHESSZp9DgbCLZsloVxvf
   

3. Get a response:

   {
     "error": {
       "code": 403,
       "message": "The caller does not have permission",
       "status": "PERMISSION_DENIED"
     }
   }