I would like to copy all files in a folder on google cloud storage bucket to another folder on the same google cloud storage bucket by python from jupyter notebook.
It is similar as my previous question: Error copying file from Linux folder to Google cloud storage bucket
But, "upload_blob" does not work for this because it is not a blob object but all files in a folder.
I need to do what "shutil.copytree()" can do on google storage bucket. I have to do this from python module so "gsutil" and "gcloud" of CLI does not work for this. I am using copy_blob at https://github.com/googleapis/python-storage/blob/main/samples/snippets/storage_copy_file.py
import sys
# [START storage_copy_file]
from google.cloud import storage
def copy_blob(
bucket_name, blob_name, destination_bucket_name, destination_blob_name,
):
"""Copies a blob from one bucket to another with a new name."""
# bucket_name = "your-bucket-name"
# blob_name = "your-object-name"
# destination_bucket_name = "destination-bucket-name"
# destination_blob_name = "destination-object-name"
storage_client = storage.Client()
source_bucket = storage_client.bucket(bucket_name)
source_blob = source_bucket.blob(blob_name)
destination_bucket = storage_client.bucket(destination_bucket_name)
# Optional: set a generation-match precondition to avoid potential race conditions
# and data corruptions. The request to copy is aborted if the object's
# generation number does not match your precondition. For a destination
# object that does not yet exist, set the if_generation_match precondition to 0.
# If the destination object already exists in your bucket, set instead a
# generation-match precondition using its generation number.
# There is also an `if_source_generation_match` parameter, which is not used in this example.
destination_generation_match_precondition = 0
blob_copy = source_bucket.copy_blob(
source_blob, destination_bucket, destination_blob_name, if_generation_match=destination_generation_match_precondition,
)
print(
"Blob {} in bucket {} copied to blob {} in bucket {}.".format(
source_blob.name,
source_bucket.name,
blob_copy.name,
destination_bucket.name,
)
)
SOURCE_BUCKET = 'MY_GCS_BUCKET'
SOURCE_BLOB = 'A_FOLDER_NAME'
DEST_BUCKET = 'MY_GCS_BUCKET'
DEST_BLOB = 'ANOTHER_FOLDER_NAME'
copy_blob(SOURCE_BUCKET, SOURCE_BLOB, DEST_BUCKET, DEST_BLOB) # error 404 POST No such object : SOURCE_BUCKET/SOURCE_BLOB
But, it does not work because it is not an object but some files. How to copy files recursively like "copytree" on GCS buckets ? thanks