3

I have already deployed a custom container in VertexAI and it is working fine, but now I would like to update my model and endpoint. I have pushed the new version of my Docker image to the Artifact Registry and I was expecting this latest version was gonna be automatically connected to VertexAI, but apparently, VertexAI is still using my old version.

How can I update the model in VertexAI automatically without having to create a new model and endpoint every time I make a new release and change the Docker image?

Javier Monsalve
  • 326
  • 3
  • 14
  • Isn't this already setup as a [customjob](https://cloud.google.com/vertex-ai/docs/training/create-custom-job) So when you push a docker image you can then just run a `customjob` command to get the model build, trained and created. Retraining also is a must if you updated the training code of your model. To my understanding. You can reuse the endpoint as it supports many models or create a new one is up to how you define your [architecture](https://cloud.google.com/vertex-ai/docs/general/deployment). – Betjens Jan 24 '22 at 17:18

2 Answers2

0

You can have more than one model running to an endpoint as described on the Reasons to deploy more than one model to the same endpoint

About the model unique name, well its something that is a nature of vertex ai. You can only have unique model IDs as show on the documentation about data model and resources.

About the Docker image event I think there are not a built-in functionality that works like that but as a workaround you can implement something similar using your notebook as you can run commands on your notebook by just using ! at the start of your code block like:

! gcloud version

About pipelines and custom job, latest samples can be found here.

Betjens
  • 1,353
  • 2
  • 4
  • 13
0

I'm in a similar boat to original questioner. I have a custom container trained model trained using docker_image:latest. When redeploying docker_image:latest to GCR, after making changes to the image's predict API, and then redeploying the model to a vertex endpoint, the changes are NOT reflected in the prediction API response.

This makes me thing that there is an internal VertexAI container registry that stores the model's image after the model has been trained and created. Then, when deploying or interacting with that model object you are pulling from the internal registry rather than the GCR one you would push to after a change to the code.

Long way of saying that I don't believe there is currently any way to make changes to any aspect of a model's docker image without rebuilding the VertexAI model object, which requires retraining. This seems like a fairly substantial limitation and I hope I'm wrong or something is released to make changes to the prediction container independent of the VertexAI model object.

mr_n0b0dy
  • 33
  • 1
  • 4
  • Hacky solution I thought of after relaunching an endpoint several times: 1) Set up a GCS bucket like `gs://current_model` 2) Every time you train a model, save the model objects to that bucket. When you retrain the same model, overwrite the current model in the bucket 3) Hard code `AIP_STORAGE_URI` in your inference code to use `gs://current_model` 4) Now you will be overwriting `AIP_STORAGE_URI` but anytime the model is relaunched you'll have the current model It's not ideal and you shouldn't overwrite `AIP_` env vars but it would work in a pinch – mr_n0b0dy Feb 22 '22 at 15:40