1

We are migrating from S3 to GCS and one of the functionalities that needs to be supported in GCS is the ability to upload large files in a multi-part fashion. S3 boto provides us with functions like initiate_multipart_upload and copy_part_from_key which I currently use for uploading large files in multiple parallel chunks.

I have seen similar discussion on stackoverflow in below two questions

Both the discussions point to this documentation which talks about an XML API to achieve this. However I'm looking for a python based implementation which uses storage.client() methods to stay consistent with rest of our integrations.

Appreciate any help on this.

Fremzy
  • 290
  • 2
  • 12

2 Answers2

1

Unfortunately the multipart upload in Cloud Storage is only available using the XML API approach.

Alternatively you could use resumable uploads, as this is the recommended way to upload large files using the GCS client.

Finally, you could also open a Feature Request in Google's Issue Tracker so that this can be considered by their product team.

Ralemos
  • 5,571
  • 2
  • 9
  • 18
0

You can just upload parts simultaneously and merge them at end.

The merging is called "Composition" at Google: https://cloud.google.com/storage/docs/composing-objects#create-composite-rest

Boto3 doesn't allow this operation.

Zulu
  • 8,765
  • 9
  • 49
  • 56