How to load a model saved in joblib file from Google Cloud Storage bucket

Question

I want to load a model which is saved as a joblib file from Google Cloud Storage bucket. When it is in local path, we can load it as follows (considering model_file is the full path in system):

loaded_model = joblib.load(model_file)

How can we do the same task with Google Cloud Storage?

score 21 · Answer 1 · answered Nov 29 '19 at 17:01

For anyone googling around for an answer to this. Here are two more options besides the obvious, to use Google AI platform for model hosting (and online predictions).

Option 1 is to use TemporaryFile like this:

from google.cloud import storage
from sklearn.externals import joblib
from tempfile import TemporaryFile

storage_client = storage.Client()
bucket_name=<bucket name>
model_bucket='model.joblib'

bucket = storage_client.get_bucket(bucket_name)
#select bucket file
blob = bucket.blob(model_bucket)
with TemporaryFile() as temp_file:
    #download blob into temp file
    blob.download_to_file(temp_file)
    temp_file.seek(0)
    #load into joblib
    model=joblib.load(temp_file)
#use the model
model.predict(...)

Option 2 is to use BytesIO like this:

from google.cloud import storage
from sklearn.externals import joblib
from io import BytesIO

storage_client = storage.Client()
bucket_name=<bucket name>
model_bucket='model.joblib'

bucket = storage_client.get_bucket(bucket_name)
#select bucket file
blob = bucket.blob(model_bucket)
#download blob into an in-memory file object
model_file = BytesIO()
blob.download_to_file(model_file)
#load into joblib
model=joblib.load(model_local)

The solution from this answer is much better than from the accepted one. — mac13k, Jul 28 '20 at 18:09
The last line in Option 2 should be ```model=joblib.load(model_file)``` — Moysey Abramowitz, Dec 07 '22 at 15:28

score 3 · Answer 2 · answered Mar 05 '20 at 10:30

3

Alternate answer as of 2020 using tf2, you can do this:

import joblib
import tensorflow as tf

gcs_path = 'gs://yourpathtofile'

loaded_model = joblib.load(tf.io.gfile.GFile(gcs_path, 'rb'))

answered Mar 05 '20 at 10:30

Kurniawan Eka Nugraha

31
1

This is a great solution if you need to load a model from GCS without having the option to write it to local disk – tb. Mar 27 '20 at 18:49
yes except that it takes like forever to read file like this. Specially if you have bulky files then this will be really slow to read files. – abggcv Apr 01 '20 at 06:58

score 2 · Answer 3 · answered Sep 13 '21 at 14:14

2

I found using gcsfs to be the fastest (and most compact) method to use:

def load_joblib(bucket_name, file_name):
    fs = gcsfs.GCSFileSystem()
    with fs.open(f'{bucket_name}/{file_name}') as f:
        return joblib.load(f)

answered Sep 13 '21 at 14:14

Eyal Shulman

699
5
15

I get permission error. 'Account _ does not have storage.objects.list access to the Google Cloud Storage bucket'. – Moysey Abramowitz Dec 07 '22 at 15:14
@MoyseyAbramowitz you should check the token parameter of GCSFileSystem constructor. I use the GOOGLE_APPLICATION_CREDENTIALS env var – Eyal Shulman Dec 07 '22 at 20:14

Iñigo · Accepted Answer · 2018-08-21T13:44:00.483

0

I don't think that's possible, at least in a direct way. I though about a workaround, but the might not be as efficient as you want.

By using the Google Cloud Storage client libraries [1] you can download the model file first, load it, and when your program ends, delete it. Of course, this means that you need to download the file every time you run the code. Here is a snippet:

from google.cloud import storage
from sklearn.externals import joblib

storage_client = storage.Client()
bucket_name=<bucket name>
model_bucket='model.joblib'
model_local='local.joblib'

bucket = storage_client.get_bucket(bucket_name)
#select bucket file
blob = bucket.blob(model_bucket)
#download that file and name it 'local.joblib'
blob.download_to_filename(model_local)
#load that file from local file
job=joblib.load(model_local)

edited Aug 21 '18 at 13:44

answered Aug 20 '18 at 13:36

Iñigo

2,500
2
10
20

so how to use ML trained models. There is only one solution in samples and it is by Pickles not Joblib, and it is only for Python 3 – Soheil Novinfard Aug 20 '18 at 14:55
I don't know if [this](https://cloud.google.com/ml-engine/docs/scikit/quickstart) is want you want or not. Either way, as it's a different question, is better to create another question asking that, as if more people have the same question as you, they can see it directly there, without having to go through this post. – Iñigo Aug 20 '18 at 15:03
Thank you, I have read this page already before, it is not saying anything about how to load joblib in this process from bucket after dumping, so the question still is remained – Soheil Novinfard Aug 20 '18 at 15:53
As I said, it is not possible to load the model from a GCS bucket. Anyway, I added a snippet of the workaround I told you about for it to be clearer – Iñigo Aug 21 '18 at 13:37
have you tested your code snippet? It has an error about reading from bytes. With pickle there are two methods for this, https://stackoverflow.com/questions/48498929/what-is-file-like-object-what-is-file-pickle-load-and-pickle-loads, but how about joblib? – Soheil Novinfard Aug 21 '18 at 14:33
yep, I worked fine for me. As you can see in the snippet, it's "model.joblib". It's the one dumped by the *Scikit-Learn* snipped found in the documentation I sent. – Iñigo Aug 21 '18 at 15:01

score 0 · Answer 5 · answered Feb 27 '21 at 13:04

For folks who are Googling around with this problem - here's another option. The open source modelstore library is a wrapper that deals with the process of saving, uploading, and downloading models from Google Cloud Storage.

Under the hood, it saves scikit-learn models using joblib, creates a tar archive with the files, and up/downloads them from a Google Cloud Storage bucket using blob.upload_from_file() and blob.download_to_filename().

In practice it looks a bit like this (a full example is here):

# Create  modelstore instance
from modelstore import ModelStore

ModelStore.from_gcloud(
   os.environ["GCP_PROJECT_ID"], # Your GCP project ID
   os.environ["GCP_BUCKET_NAME"], # Your Cloud Storage bucket name
)

# Train and upload a model (this currently works with 9 different ML frameworks)
model = train() # Replace with your code to train a model
meta_data = modelstore.sklearn.upload("my-model-domain", model=model)

# ... and later when you want to download it
model_path = modelstore.download(
  local_path="/path/to/a/directory",
  domain="my-model-domain",
  model_id=meta_data["model"]["model_id"],
)

The full documentation is here.

score 0 · Answer 6 · answered Jun 18 '22 at 14:02

0

This is the shortest way I found so far:

from google.cloud import storage

client = storage.Client()
bucket = client.get_bucket("my-gcs-bucket")
blob = bucket.blob("model.joblib")
with blob.open(mode="rb") as file:
    model = joblib.load(file)

answered Jun 18 '22 at 14:02

Matheus Couto

61
1
2

How to load a model saved in joblib file from Google Cloud Storage bucket

6 Answers6

Linked