3

I am trying to run a Custom Training Job to deploy my model in Vertex AI directly from a Jupyterlab. This Jupyterlab is instantiated from a Vertex AI Managed Notebook where I already specified the service account.

My aim is to deploy the training script that I specify to the method CustomTrainingJob directly from the cells of my notebook. This would be equivalent to pushing an image that contains my script to container registry and deploying the Training Job manually from the UI of Vertex AI (in this way, by specifying the service account, I was able to corectly deploy the training job). However, I need everything to be executed from the same notebook.

In order to specify the credentials to the CustomTrainingJob of aiplatform, I execute the following cell, where all variables are correctly set:

import google.auth
from google.cloud import aiplatform
from google.auth import impersonated_credentials

source_credentials = google.auth.default()
target_credentials = impersonated_credentials.Credentials(
source_credentials=source_credentials,
target_principal='SERVICE_ACCOUNT.iam.gserviceaccount.com',
target_scopes = ['https://www.googleapis.com/auth/cloud-platform'])

aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_NAME)

job = aiplatform.CustomTrainingJob(
    display_name=JOB_NAME,
    script_path=SCRIPT_PATH,
    container_uri=MODEL_TRAINING_IMAGE,
    credentials=target_credentials
)

When after the job.run() command is executed it seems that the credentials are not correctly set. In particular, the following error is returned:

/opt/conda/lib/python3.7/site-packages/google/auth/impersonated_credentials.py in _update_token(self, request)
    254 
    255         # Refresh our source credentials if it is not valid.
--> 256         if not self._source_credentials.valid:
    257             self._source_credentials.refresh(request)
    258 

AttributeError: 'tuple' object has no attribute 'valid'

I also tried different ways to configure the credentials of my service account but none of them seem to work. In this case it looks like the tuple that contains the source credentials is missing the 'valid' attribute, even if the method google.auth.default() only returns two values.

Megapiro
  • 53
  • 5

2 Answers2

0

To run the custom training job using a service account, you could try using the service_account argument for job.run(), instead of trying to set credentials. As long as the notebook executes as a user that has act-as permissions for the chosen service account, this should let you run the custom training job as that service account.

JankyJ
  • 11
  • 3
0

Using service account credentials from details stored as a Json file.

  1. create a credential.json from your service account (add file to .gitignore)
  2. load Service Account file as an object with
from google.oauth2 import service_account

credentials = service_account.Credentials.from_service_account_file(credentials.json)
  1. initialize aiplatform instance
aiplatform.init(project=project, location=location, credentials=credentials)

Note that this question doesn't mention neither Kubeflow nor Pipeline, if Kubeflow sdk is needed, the authentication with aiplatform is not valid or enough for kfp.

Alsushi
  • 373
  • 3
  • 14