0

I'm using CloudBerry Explorer to copy files from one bucket to another bucket. both buckets are under the same project on Google Cloud Platform. I can easily copy small size files but when trying to copy 1GB+ files from the same path to the other, I get an error:

"The remote server returned an error: (500) Internal Server Error.

Permissions are correct since moving small size files is fine.

What could be the issue?

CloudBerry Log:

2020-12-09 14:23:52,298 [Google] [9] INFO  - InternalGetObjectExists start, bucket: DEDUCTED-BUCKET, key DEDUCTED-FILENAME.mp4
2020-12-09 14:23:52,781 [Google] [9] INFO  - InternalGetObjectExists ready, bucket: DEDUCTED-BUCKET, key DEDUCTED-FILENAME.mp4, status: 404
2020-12-09 14:23:52,782 [Google] [9] INFO  - InternalGetObjectExists start, bucket: DEDUCTED-BUCKET, key DEDUCTED-PATH/DEDUCTED-FILENAME.mp4
2020-12-09 14:23:52,992 [Google] [9] INFO  - InternalGetObjectExists ready, bucket: DEDUCTED-BUCKET, key DEDUCTED-PATH/DEDUCTED-FILENAME.mp4, status: 200
2020-12-09 14:23:52,992 [Google] [9] INFO  - InternalCopyObjectNonChunked start, src bucket: DEDUCTED-BUCKET, src key: DEDUCTED-PATH/DEDUCTED-FILENAME.mp4, dst bucket: DEDUCTED-BUCKET, dst key: DEDUCTED-FILENAME.mp4
2020-12-09 14:25:17,510 [Google] [9] INFO  - InternalCopyObjectNonChunked ready, src bucket: DEDUCTED-BUCKET, src key: DEDUCTED-PATH/DEDUCTED-FILENAME.mp4, dst bucket: DEDUCTED-BUCKET, dst key: DEDUCTED-FILENAME.mp4, status: 500
2020-12-09 14:25:17,510 [Google] [9] ERROR - Http response status: 500: Internal Server Error
2020-12-09 14:25:17,511 [Google] [9] ERROR - <?xml version='1.0' encoding='UTF-8'?><Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><Details>AFfi+BFP7TevpPSJUbrZ2g1VXu3cm/uZHLEPxDMKeQro1D9stEU/xcIHUePCm9meA3u2N+gGUX41/UHUd1WIDNA+tOjFrbNRZsC2T8twlFOBURzaHcO5GfU=</Details></Error>
2020-12-09 14:25:17,511 [Google] [9] ERROR - InternalCopyObjectNonChunked failed. src bucket: DEDUCTED-BUCKET, src key: DEDUCTED-PATH/DEDUCTED-FILENAME.mp4, dst bucket: DEDUCTED-BUCKET, dst key: DEDUCTED-FILENAME.mp4
CloudBerryLab.Base.Exceptions.Status500Exception
The remote server returned an error: (500) Internal Server Error.

System.Net.WebException
The remote server returned an error: (500) Internal Server Error.
   at System.Net.HttpWebRequest.GetResponse()
   at gZ.A(HB , Action1` , HttpWebRequest , HS )

2020-12-09 14:25:17,513 [CL] [9] ERROR - Command::Run failed:
Copy; Source:DEDUCTED-PATH/DEDUCTED-FILENAME.mp4; Destination:/DEDUCTED-BUCKET/
CloudBerryLab.Base.Exceptions.Status500Exception: The remote server returned an error: (500) Internal Server Error. ---> System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
   at System.Net.HttpWebRequest.GetResponse()
   at gZ.A(HB , Action1` , HttpWebRequest , HS )
   --- End of inner exception stack trace ---
   at Tm.a(String , String , String , String , String , MetadataDirective , Gw , Jt )
   at Tm.A(String , String , String , String , String , String , Gw )
   at sU.A(Tm , sU , Gw , lD )
   at sU.BV(lL , String , lD )
   at lF.RunInternal()
   at SE.RunInternal()
   at lE.fa()
Brian Tompsett - 汤莱恩
  • 5,753
  • 72
  • 57
  • 129
Tal Malka
  • 1
  • 1
  • Which component raise the error? Cloud Storage or CloudBerry? I don't know CloudBerry but I think it's on this side. – guillaume blaquiere Dec 09 '20 at 12:29
  • I use Cloudberry Explorer. I copy multi-gigabyte files very often (VMDKs and VHDs). I do not have a problem. Edit your question showing the API request and the response and the relevant Cloudberry log entries. – John Hanley Dec 09 '20 at 18:11
  • Hi Tal, Welcome to SO. John is right, I have used Cloudberry before and I've never got an error like you described. In order to can help you, it could be better if you share the Cloudberry log entries. – Samuel Romero Dec 09 '20 at 22:59
  • Hi guys, thank you for replying. I added the log to the question. – Tal Malka Dec 10 '20 at 08:55
  • I don't know Cloudberry, but this error is strange "InternalCopyObjectNonChunked". Do you have some parameters to change to copy file differently? – guillaume blaquiere Dec 10 '20 at 14:44
  • The log entries do not show any relevant information. I have tried at my end again from different computers and I was able to copy the files from a bucket to another. I would recommend checking if you have some configuration in your computar that can cause this issue. – Samuel Romero Dec 16 '20 at 23:48

1 Answers1

0

I encountered the same issue recently. Initially, the issue was reported with a Talend job using the tGCSCopy component failing with a 500 error. It was also experienced with Cloudberry with manual testing. Works fine with small files, but larger ones are where the issue happens. Turns out the issue is related to bucket to bucket copies where the target bucket is using a customer managed encryption key for encryption. Standard Google managed encryption does not have the same issue and using Google provided utils like gsutil do not exhibit the same behavior.

I also tested this with a source object encrypted with the same key and didn't make a difference. Talend uses jets3t, an s3 compliant library to interact with Cloud Storage, and I'm assuming Cloudberry is doing something similar that doesn't play nice with this type of scenario.

No resolution as of now, Google is likely going to say don't use 3rd party products.

Brian Adamonis
  • 1