Secret Manager from Vertex AI Pipeline

Question

I am working in GCP creating a Vertex AI pipeline with kubeflow and it is time for me to store my API keys more securely. I am very new to GCP and unfamiliar with the environment so I've been trying to follow a few tutorials but have hit a roadblock. I want to store my secrets in Secret Manager and then later access them from the pipeline I've written. I have no problem creating secrets and viewing them in the GUI but when it comes to compiling my pipeline i get the error: google.api_core.exceptions.PermissionDenied: 403 Permission denied on resource project...

So it seems that the account running my pipelines does not have access to the secrets I have created. My question is then, how do I check which account is running the pipeline so I can grant it access? Or is there really another underlying problem here?

Code trying to access the secret:

 client = secretmanager.SecretManagerServiceClient()
 secret_name = "secret_name"
 request = {'name': f"path/{secret_name}/versions/latest"}
 response = client.access_secret_version(request)
 secret_string = response.payload.data.decode("UTF-8")

EDIT: I can add that I have been playing around a lot with account permissions but my best guess is that the account that is found under Vertex AI>Workbench>the notebook I am using's notebook details>Service account is the one that needs permission. Is this not it?

Answer 1

In my case, the problem was that I was not using a custom service account, as according to this page:

Note: If you want your custom training code to obtain an OAuth 2.0 access token with the https://www.googleapis.com/auth/cloud-platform scope, then you must use a custom service account for training. You cannot give this level of access to the Vertex AI Custom Code Service Agent.

As the secret manager client requires the aforementioned access token, you will need to create your own custom service account first. After you create one (with proper permission attached, namely, Secret Manager Secret Accessor), you can attach the service account to the job that needs it.

 @component(
     packages_to_install=['google-cloud-secret-manager']
 )
 def print_secret_op(project_id: str, secret_id: str, version_id: str) -> str:
     from google.cloud import secretmanager

     secret_client = secretmanager.SecretManagerServiceClient()
     secret_name = f'projects/{project_id}/secrets/{secret_id}/versions/{version_id}'
     response = secret_client.access_secret_version(request={"name": secret_name})
     payload = response.payload.data.decode("UTF-8")
     answer = "The secret is: {}".format(payload)
     print(answer)
     return answer


print_secret_job = create_custom_training_job_from_component(
    print_secret_op,
    service_account="your-service-account-email",
)

Answer 2

Running the command gcloud auth list might help you to determine the account you are using. Additionally, you can troubleshoot the accounts that you have created in the project and see the roles that the accounts have by following this documentation.

To access secrets with Secret Manager using a pipeline, you need to grant the service account that runs the pipeline with secret manager permission. You can see how to configure a service account with granular permissions section of Configure your Google Cloud project for Vertex AI pipelines.

After setting the secret manager permissions on the service account that is running the pipeline you can access the Secrets.

Additionally, you can check this documentation to see how to access secrets with a kubeflow pipeline.

Answer 3

Possible Cause

The error google.api_core.exceptions.PermissionDenied: 403 Permission denied in Vertex Pipeline components is typically because you did not specify the project to the GCP client library instance.

client = secretmanager.SecretManagerServiceClient()  # <---- NO project specified

It is explained in Access Google Cloud services in your code.

This problem occurs because Vertex AI does not run your code directly in your Google Cloud project. Instead, Vertex AI runs your code in one of several separate projects managed by Google. Vertex AI uses these projects exclusively for operations related to your project. Therefore, don't try to infer a project ID from the environment in your training or prediction code; specify project IDs explicitly.

Because you did not specify the project ID to the SecretManagerServiceClient, it tried to connect to the Secret Manager service of the Vertex AI project managed by Google, which will not allow you to access it, of course.

Fix

As per the example in Access Google Cloud services in your code, specify YOUR project ID to the project parameter when instantiating a GCP service client.

import os

from google.cloud import bigquery

project_number = os.environ["CLOUD_ML_PROJECT_ID"]

client = bigquery.Client(project=project_number)