Instance Group won't create an instance with GPU: Not enough resources

Question

I'm able to create an instance with NVIDIA-K80 manually, however my instance group shows a warning on the instance:

Instance 'instance-6lqk' creation failed: The zone 'projects/my-project/zones/us-central1-a' does not have enough resources available to fulfill the request. Try a different zone, or try again later.

Note: Both are create in the same zone

Did you try changing the zone? – Maciek Semik Dec 03 '18 at 13:50 — Maciek Semik, Dec 03 '18 at 13:50
Maybe change the cloud provider! I can't change the zone. – aclowkay Dec 03 '18 at 13:54 — aclowkay, Dec 03 '18 at 13:54

score 2 · Answer 1 · answered Dec 11 '18 at 22:20

(Works for Google)

The error message you received indicated that you did everything right but the zone couldn't fulfill your request. From time to time, this happens for a variety of reasons in a single zone. My suggestion would be to use multiple zones and/or multiple regions so that if this happens to you in one zone, you can simply create capacity in another zone.

Note, a lot of our Preemptible GPU users looking to run large workloads on many GPUs do just this. Ask for quota in many regions and run multi-region instance groups to have the best chance to get access to the most capacity possible.

score 1 · Answer 2 · answered Dec 03 '18 at 14:07

I created my instance group from an instance templates from the Google doc

Using the same example from the Google Doc below:

gcloud beta compute instance-templates create gpu-template \
--machine-type n1-standard-2 \
--boot-disk-size 250GB \
--accelerator type=nvidia-tesla-k80,count=1 \
--image-family ubuntu-1604-lts --image-project ubuntu-os-cloud \
--maintenance-policy TERMINATE --restart-on-failure \
--metadata startup-script='#!/bin/bash
echo "Checking for CUDA and installing."
# Check for CUDA and try to install.
if ! dpkg-query -W cuda-9-0; then
  curl -O http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_9.0.176-1_amd64.deb
  dpkg -i ./cuda-repo-ubuntu1604_9.0.176-1_amd64.deb
  apt-get update
  apt-get install cuda-9-0 -y
fi'

However I found some recommendations when you are creating a GPU instances. Make sure you have necessary Quotas in the zone, and Michael commented about the GPU restrictions.

hope it can be useful to you.

Instance Group won't create an instance with GPU: Not enough resources

2 Answers2