4

When using TPUs on Google Colab (such as in the MNIST example), we are told to create a GCS bucket. However, it doesn't tell us where. Without knowing the region/zone of the Colab instance, I am afraid to create a bucket in fear of running into billing issues.

There are actually several questions:

  1. Is accessing a GCS bucket from Colab free, or do the normal network egress fees apply?
  2. Can I get the region/zone of the colab instance? Most likely not.
  3. If the question to both questions above is "no": is there any solution for minimizing costs when using TPUs with Colab?
Bob Smith
  • 36,107
  • 11
  • 98
  • 91
David Nemeskey
  • 640
  • 1
  • 5
  • 16

2 Answers2

5

Thank you for your question.

No, you can not get the region/zone of the colab instance. So you can try creating a multi-regional GCS bucket which should be accessible by the colab. As per the comment, https://github.com/googlecolab/colabtools/issues/597#issuecomment-502746530, Colab TPU instances are only in US zone. So while creating a GCS bucket, you can choose a Multi-region bucket in US.

Checkout https://cloud.google.com/storage/pricing to get more details about the pricing information for the GCS buckets.

You can also sign up for a Google Cloud Platform account with 5GB of free storage and $300 in credits at https://cloud.google.com/free/, so that should be able to provide you with enough credits to get started.

Ruben Helsloot
  • 12,582
  • 6
  • 26
  • 49
aman2930
  • 275
  • 2
  • 9
  • I upvoted you; however, maybe I am missing something, but a multi-regional bucket doesn't seem to solve the problem if e.g. the Colfab instance is in the US while the bucket is in the EU. – David Nemeskey Jun 09 '20 at 21:24
  • As per the comment, https://github.com/googlecolab/colabtools/issues/597#issuecomment-502746530, Colab TPU instances are only in US zone. So while creating a GCS bucket, you can choose a Multi-region bucket in US. I hope that should resolve the issue. – aman2930 Jun 09 '20 at 21:55
  • Note also that multi-region buckets are more expensive than single-region buckets. So this might not be the cheapest option depending on the size of your data and frequency of your requests. If you have small data you plan to access (very) frequently this might be a good option, but if you have lots of data to store and you're only accessing it infrequently the egress charges could be less signficant than the added storage costs. – Michael Delgado Jan 06 '22 at 20:55
0

We are told to create a GCS bucket. However, it doesn't tell us where.

Running (within Colab)

!curl ipinfo.io

You get something similar to

{
  "ip": "3X.20X.4X.1XX",
  "hostname": "13X.4X.20X.3X.bc.googleusercontent.com",
  "city": "Groningen",
  "region": "Groningen",
  "country": "NL",
  "loc": "53.21XX,6.56XX",
  "org": "AS396XXX Google LLC",
  "postal": "9711",
  "timezone": "Europe/Amsterdam",
  "readme": "https://ipinfo.io/missingauth"
}

Which tells you where you Colab is running.

You can create a GCS bucket in just one region (if you don't need multi-region).

Assuming you don't change country/area very often, you can check that a few times (different days) and get an idea of where your Colab is probably going to be located.

For your other questions (egress,...) see the Conclusion on https://ostrokach.gitlab.io/post/google-colab-storage/

[...] Google Cloud Storage is a good option for hosting our data. Only we should be sure to check that the Colab notebook is running in the same continent as your Cloud Storage bucket, or we will incur network egress charges!

Rub
  • 2,071
  • 21
  • 37