CloudBerry PowerShell Multipart

Question

I've written a powershell script to upload from a windows system to an Amazon S3 Bucket. The script successfully uploads all files, except those over 5Gb. I have the Cloud Berry Explorer Pro license which allows for multipart upload on files up to 5TB. However there is no flag for multipart on the powershell snapin documentation. Cloudberry support directed me here as they only support the GUI not the powershell snapin. When running my script I get the error

"WARNING: Your proposed upload exceeds the maximum allowed object size (5 Gb)".

So question is, does anyone know if there is a command line option, or another way to enable multipart upload to Amazon S3 using Cloudberry Explorer Pro's Powershell Snapin?

score 0 · Accepted Answer · answered May 24 '17 at 08:51

0

Set-CloudOption UseChunks=true

answered May 24 '17 at 08:51

Anton Zorin

371
2
13

Thanks Antoine, this was the correct answer to get around the 5GB upload limit in powershell. We wound up using this as documented in my update below. – TITECHSD Jun 08 '17 at 14:46

score 0 · Answer 2 · answered Jun 07 '17 at 13:08

I'm looking for the same in Powershell.

I believe in GUI, the original chunking mechanism has been depreciated. I have not tested myself, but I assume Powershell option UseChunks=true is still using the old mechanism? If so, files may be split into multiple parts and not automatically recombined when they arrive on S3. The new GUI Multipart Upload facility sorts this all out for you.

Annoyed Cloudberry still advertise Powershell as a component of Explorer (Free & Pro), but don't support it, even for fully paid up PRO support customers.

score 0 · Answer 3 · edited Jun 20 '20 at 09:12

We did purchase the cloudberry explorer pro license for the native multipart upload capability, but we wanted to automate it. I believe based on their documentation that the old chunk method is deprecated in favor of their new multi-part functionality. We wound up testing the options listed in the powershell documentation. Those options are as follows:

Set-CloudOption -UseChunks -ChunkSizeKB

"Defines a size of chunk in KB; files larger than a chunk will be divided into chunks."

We verified that this was successfully uploading files beyond the 5GB restriction to our S3 bucket. I attempted to get a response from Cloudberry as to whether this was the old Chunking method or the new Multi-Part method, but I was unable to get a straight answer. They confirmed that because we were using pro, this powershell option was supported, but they failed to confirm which option the powershell command was using.

From what I can tell it appears that Cloudberry's legacy chunking mechanism would actually just break the file into individual files and thus would appear in S3 as multiple files. The Chunk Transparency mechanism in Cloudberry Explorer would allow the multiple chunks to appear as a single file in the Cloudberry Explorer GUI only. Since I can see the file as a single file on the S3 side, I'm assuming that the powershell option uses the new Multi-Part functionality and not the legacy Chunking functionality. Again I was not able to confirm this through Cloudberry so it's speculation on my part.

However, I can confirm that using the powershell options above will get you around the 5GB upload limit when using powershell.

CloudBerry PowerShell Multipart

3 Answers3