Use ADC Inside Container From Google Batch Job Including Scopes

Question

I trying out Google's Batch service and spinning up a compute engine vm for a small python script. I am executing this by referring to a pre-build docker image and would like to access the default application credentials within that docker instance. Though it seems I just cannot access the default application credentials.

Is there something I am doing wrong?

Here is my job.json:

{
    "taskGroups": [
      {
        "taskSpec": {
          "runnables": [
            {
              "container": {
                "imageUri": "europe-west3-docker.pkg.dev/project-id/containers/service:v1"
              }
            }
          ]
        }
      }
    ],
    "allocationPolicy": {
        "serviceAccount": {
        "email": "sa-test@some-project.iam.gserviceaccount.com"
        }
    },
    "logsPolicy": {"destination": "CLOUD_LOGGING"}
}

Here is my Dockerfile:

FROM python:3.10
WORKDIR /app
ENV PATH $PATH:$HOME/.local/bin
COPY . .
RUN pip install --no-cache-dir --upgrade -r /app/requirements.txt
ENTRYPOINT ["python", "main.py"]

And here my main.py file:

import google.auth

credentials, _ = google.auth.default()
print(f"Service account email: {credentials.service_account_email}")

prints: Service account email: default

I already tested accessing the GOOGLE_APPLICATION_CREDENTIALS environment inside the main.py file but it is not set. When manually setting it with a key.json file it works but this is something I am trying to avoid.

Default is not bad!! Did you try to generate an access token or to access a service? — guillaume blaquiere, Mar 20 '23 at 19:44

score 1 · Answer 1 · answered Mar 21 '23 at 16:35

After much sweat and pain I have finally come around. Also with the help of @guillaume blaquiere ..that default confused me a lot. Thanks for clarifying that that was not the problem :)

I was trying to use the drive client which means that when using a service account I need to add scopes. In the python code I did the following:

import google.auth
from googleapiclient.discovery import build

SCOPES = ["https://www.googleapis.com/auth/drive"]
credentials, _ = google.auth.default(scopes=SCOPES)
drive_client = build("drive", "v3", credentials=credentials)

Though that was not enough. I also needed to add the scopes to the VM instance configuration. Here is my updated working code:

{
"taskGroups": [
  {
    "taskSpec": {
      "runnables": [
        {
          "container": {
            "imageUri": "europe-west3-docker.pkg.dev/project-id/containers/service:v1"
          }
        }
      ]
    }
  }
],
"allocationPolicy": {
    "serviceAccount": {
       "email": "sa-test@some-project.iam.gserviceaccount.com",
       "scopes": ["https://www.googleapis.com/auth/drive"]
    }
},
"logsPolicy": {"destination": "CLOUD_LOGGING"}
}

here is what got me the hint: https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances#changeserviceaccountandscopes

Use ADC Inside Container From Google Batch Job Including Scopes

1 Answers1