How to upload folder from local to GCP bucket using python

Question

I am following this link and getting some error:

How to upload folder on Google Cloud Storage using Python API

I have saved model in container environment and from there I want to copy to GCP bucket.

Here is my code:

storage_client = storage.Client(project='*****')
def upload_local_directory_to_gcs(local_path, bucket, gcs_path):

   bucket = storage_client.bucket(bucket)

    assert os.path.isdir(local_path)
    for local_file in glob.glob(local_path + '/**'):
        
        print(local_file)


        
        print("this is bucket",bucket)
        blob = bucket.blob(gcs_path)
        print("here")
        blob.upload_from_filename(local_file)
        print("done")

path="/pythonPackage/trainer/model_mlm_demo" #this is local absolute path where my folder is. Folder name is **model_mlm_demo**
buc="py*****" #this is my GCP bucket address
gcs="model_mlm_demo2/" #this is the new folder that I want to store files in GCP

upload_local_directory_to_gcs(local_path=path, bucket=buc, gcs_path=gcs)

/pythonPackage/trainer/model_mlm_demo has 3 files in it config, model.bin and arguments.bin`

ERROR

The codes doesn't throw any error, but there is no files uploaded in GCP bucket. It just creates empty folder.

Can you share a screenshot of your bucket and empty folder from GCP Storage? — Priyashree Bhadra, Mar 25 '22 at 11:07

score 1 · Answer 1 · answered Mar 25 '22 at 10:55

What I can see the error is, you don't need to pass the gs:// as the bucket parameter. Actually, here is an example you may need to check out,

https://cloud.google.com/storage/docs/uploading-objects#storage-upload-object-python

def upload_blob(bucket_name, source_file_name, destination_blob_name):
    """Uploads a file to the bucket."""
    # The ID of your GCS bucket
    # bucket_name = "your-bucket-name"
    # The path to your file to upload
    # source_file_name = "local/path/to/file"
    # The ID of your GCS object
    # destination_blob_name = "storage-object-name"

    storage_client = storage.Client()
    bucket = storage_client.bucket(bucket_name)
    blob = bucket.blob(destination_blob_name)

    blob.upload_from_filename(source_file_name)

    print(
        "File {} uploaded to {}.".format(
            source_file_name, destination_blob_name
        )
    )

Even after I remove gs, I cannot see any files in folder. Strange. It's almost same code I have written above? — MAC, Mar 25 '22 at 10:59
@MAC I noticed you pas the directory to the `upload_from_filename` but you should pass file paths one by one, but not the directory path. — Zhong Dai, Mar 26 '22 at 00:34

score 1 · Answer 2 · answered Mar 25 '22 at 15:17

I have reproduced your issue and the below code snippet works fine. I have updated the code based on folders and names you have mentioned in the question. Let me know if you have any issues.

import os
import glob
from google.cloud import storage
storage_client = storage.Client(project='')

def upload_local_directory_to_gcs(local_path, bucket, gcs_path):

    bucket = storage_client.bucket(bucket)

    assert os.path.isdir(local_path)
    for local_file in glob.glob(local_path + '/**'):

        print(local_file)

        print("this is bucket", bucket)
        filename=local_file.split('/')[-1]
        blob = bucket.blob(gcs_path+filename)
        print("here")
        blob.upload_from_filename(local_file)
        print("done")


# this is local absolute path where my folder is. Folder name is **model_mlm_demo**
path = "/pythonPackage/trainer/model_mlm_demo"
buc = "py*****"  # this is my GCP bucket address
gcs = "model_mlm_demo2/"  # this is the new folder that I want to store files in GCP

upload_local_directory_to_gcs(local_path=path, bucket=buc, gcs_path=gcs)

Tsvi Sabo · Answer 3 · 2022-08-28T20:38:23.953

1

I just came across the gcsfs library which seems to be also about better interfaces

You could copy an entire directory into a gcs location like this:


def upload_to_gcs(src_dir: str, gcs_dst: str):
    fs = gcsfs.GCSFileSystem()
    fs.put(src_dir, gcs_dst, recursive=True)

edited Aug 28 '22 at 20:38

answered Aug 28 '22 at 16:10

Tsvi Sabo

575
2
11

score 0 · Answer 4 · answered Mar 25 '22 at 18:16

I figured out a way using subprocess to upload model artefacts in GCP bucket.

import subprocess

subprocess.call('gsutil cp -r source_folder_in_local gs://*****/folder_name', shell=True, stdout=subprocess.PIPE)

If gsutil is not installed. You can install using this link:

https://cloud.google.com/storage/docs/gsutil_install

How to upload folder from local to GCP bucket using python

4 Answers4