3

In the context of a personal project, Im trying to use Vertex AI to run a TFX pipeline to train a model using custom training, based on this guide. When I run the pipeline I get the error:

com.google.cloud.ai.platform.common.errors.AiPlatformException: code=RESOURCE_EXHAUSTED, message=The following quota metrics exceed quota limits: aiplatform.googleapis.com/custom_model_training_cpus

On the IAM quotas I have limit "1" for the resource "Custom model training CPUs for N1/E2 machine types per region", for all regions, and 0% current usage for each one of them. I even tried multiple regions and multiple types of machines (n1, e2, ...) and I alway get that quota limit error.

Can anyone explain why Im getting this quota error?

theeDude
  • 107
  • 1
  • 8
  • You can contact [Cloud Billing Support](https://cloud.google.com/support/billing). – Jose Gutierrez Paliza Aug 22 '22 at 18:13
  • @JoseGutierrezPaliza Already did that. I only have basic support plan, so it was pretty much useless. Please dont tell me that users need to subscribe to a paid support plan in order just to try Vertex AI. – theeDude Aug 26 '22 at 14:39
  • 1
    Same here. Since all machine types require at least 4 CPUs, there is just no way to use Vertex AI without paying for a support plan upgrade only to request an increase on quota limits. That is messed up to say the least. – Nicky Aug 01 '23 at 09:22

1 Answers1

0

On the IAM quotas I have limit "1" for the resource "Custom model training CPUs for N1/E2 machine types per region", for all regions, and 0% current usage for each one of them.

The default quotas for aiplatform.googleapis.com/custom_model_training_cpus are listed here, and their range is between 20 and 2,200, depending on the region. I'm not sure why your limit would be 1, but I believe that means you will not be able to use any machine types that use more than one vCPU, so you can't even use "n1-standard-2", even if your pipeline would only use one machine of type "n1-standard-2" for training.

One thing you can try is editing the quotas for your GCP project from the quotas page by selecting the quota(s) in the table and then clicking the "Edit quotas" button.

JankyJ
  • 11
  • 3
  • Thanks. I could not edit any quotas, I would have to pay for support just to even try to request for higher quotas. I simply gave up using GCP and moved to other and much better cloud provider. – theeDude Oct 15 '22 at 09:58