0

I would like to copy all files in a folder on google cloud storage bucket to another folder on the same google cloud storage bucket by python from jupyter notebook.

It is similar as my previous question: Error copying file from Linux folder to Google cloud storage bucket

But, "upload_blob" does not work for this because it is not a blob object but all files in a folder.

I need to do what "shutil.copytree()" can do on google storage bucket. I have to do this from python module so "gsutil" and "gcloud" of CLI does not work for this. I am using copy_blob at https://github.com/googleapis/python-storage/blob/main/samples/snippets/storage_copy_file.py

import sys

# [START storage_copy_file]
from google.cloud import storage


def copy_blob(
    bucket_name, blob_name, destination_bucket_name, destination_blob_name,
):
    """Copies a blob from one bucket to another with a new name."""
    # bucket_name = "your-bucket-name"
    # blob_name = "your-object-name"
    # destination_bucket_name = "destination-bucket-name"
    # destination_blob_name = "destination-object-name"

    storage_client = storage.Client()

    source_bucket = storage_client.bucket(bucket_name)
    source_blob = source_bucket.blob(blob_name)
    destination_bucket = storage_client.bucket(destination_bucket_name)

    # Optional: set a generation-match precondition to avoid potential race conditions
    # and data corruptions. The request to copy is aborted if the object's
    # generation number does not match your precondition. For a destination
    # object that does not yet exist, set the if_generation_match precondition to 0.
    # If the destination object already exists in your bucket, set instead a
    # generation-match precondition using its generation number.
    # There is also an `if_source_generation_match` parameter, which is not used in this example.
    destination_generation_match_precondition = 0

    blob_copy = source_bucket.copy_blob(
        source_blob, destination_bucket, destination_blob_name, if_generation_match=destination_generation_match_precondition,
    )

    print(
        "Blob {} in bucket {} copied to blob {} in bucket {}.".format(
            source_blob.name,
            source_bucket.name,
            blob_copy.name,
            destination_bucket.name,
        )
    )

SOURCE_BUCKET = 'MY_GCS_BUCKET'
SOURCE_BLOB = 'A_FOLDER_NAME'
DEST_BUCKET = 'MY_GCS_BUCKET'
DEST_BLOB = 'ANOTHER_FOLDER_NAME'
copy_blob(SOURCE_BUCKET, SOURCE_BLOB, DEST_BUCKET, DEST_BLOB) # error 404 POST No such object : SOURCE_BUCKET/SOURCE_BLOB

But, it does not work because it is not an object but some files. How to copy files recursively like "copytree" on GCS buckets ? thanks

user3448011
  • 1,469
  • 1
  • 17
  • 39

0 Answers0