0

After getting frustrated with Azure, I decided to try GCP. I wanted to try training a deep learning image classification model using GCP. To start off, I went to Cloud Storage through the Google Cloud Console UI and made a bucket to upload my images and... it freezes when I try to upload my image folder. Not a good first experience.

I've tried half a dozen times. I always get the little popup "upload started," but then it freezes. If I close it and reopen, the bucket will have made the folder and downloaded 3 or so images, but even if I let it run all night, it never makes progress beyond that. It just stops uploading and becomes nonresponsive. I have about 38,000 images, in total the folder size that I'm trying to upload is 2.5 GB.

There is no error message, and no sign that anything went wrong when I reload aside from the fact that only 3 or so images downloaded. Nothing else on my computer freezes. If I have other tabs open in the browser, they do not freeze. The tab with Google Console in it simply silently freezes. I tried Data Transfer to move the files to Google directly from Azure Blob Storage, but Azure wouldn't give Google permission to access the storage.

Can someone recommend a possible work-around to this? Should I be uploading my files some other way?

JaysonM
  • 596
  • 1
  • 10
AJV
  • 153
  • 8

3 Answers3

1

Whether or not a large number of files can be uploaded via the Console in-browser appears to be a bit sporadic. In my particular case, attempting to upload a large folder (not large files in a folder, a large NUMBER of files in a folder) at once results in the browser freezing due to memory problems in GCP's Cloud Storage AND in AWS's S3 when I try to upload it through their browser portals. I have tried this on Windows 10, on Linux Mint, and tried on both Google Chrome and Firefox to no avail.

Some GCP and AWS users report freezing or crashes when the number of files in the folder is large, but not necessarily when the size of the folder itself is large. This is my experience exactly--I can upload a large file or a folder with a few hundred files through Cloud Console, but not a folder with many thousands of files.

The only stable solution I've found is to surrender and abandon trying to upload a massive number of files in a folder via browser portals and instead upload via CLI. It isn't the solution I'd hoped for, but it is the only one I've found.

EDIT:

Since this correct (and only working) solution was downvoted for some reason,

See the comments to Kainax's answer to this question for multiple AWS users experiencing this problem.

See the question and answers to this question for the same issue I had in GCP.

Since this is an issue and not a feature, there is no official documentation on it.

Neither I nor anyone else as yet could solve this through the Cloud Console and this has been a recurring issue for years with some users. The only clear workaround is to avoid using the Console UI altogether when uploading a folder with a large number of files.

AJV
  • 153
  • 8
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community May 13 '22 at 17:51
0

Tried to recreate image folder with 2G+ size and upload it on bucket using console. Below steps are working on my end.

  1. Create Image files up to 2G+
  2. Cloud Storage Browser > Select Bucket
  3. Upload Folder > Select Folder > Upload
  4. On/Off Show Deleted data on Bucket Details Page
  5. Validate on Cloud Shell via gsutil command.
gsutil ls
gsutil du -h -a gs://$BUCKETNAME
gsutil ls gs://$BUCKETNAME/* | wc -l

At first upon uploading, not responding pop up appear. Tried to open new tab and try to check my bucket on console. Also I can see only 3 files upon checking the folder on the bucket. I give it some time for about 15 mins and tried to refresh still 3 Files on the folder. Tried to on and off the Show deleted data (Middle Right part of Bucket Details) and some of the files are showing. I Give it a little more time like 30 - 45 mins and it was completed.

==> standard input <==
3.02 KiB     gs://testupload/testimage/download (992).jpeg#1651637939860508
3.02 KiB     gs://testupload/testimage/download (993).jpeg#1651637579558398
3.02 KiB     gs://testupload/testimage/download (994).jpeg#1651637761564156
3.02 KiB     gs://testupload/testimage/download (995).jpeg#1651637080592200
3.02 KiB     gs://testupload/testimage/download (996).jpeg#1651637475578587
3.02 KiB     gs://testupload/testimage/download (997).jpeg#1651637634800927
3.02 KiB     gs://testupload/testimage/download (998).jpeg#1651637897219133
3.02 KiB     gs://testupload/testimage/download (999).jpeg#1651638427607588
3.02 KiB     gs://testupload/testimage/download.jpeg#1651636931991159
2.26 GiB     gs://testupload/testimage/

XXXXXXXXX@cloudshell:~ (XXXXXXX)$ gsutil ls gs://testupload/* | wc -l
4564

Source File image

Uploaded Files on Console

JaysonM
  • 596
  • 1
  • 10
  • I tried opening a new tab and toggling "show deleted data" on and off, but it still never uploaded more than 5 images. It immediately uploaded those 5, then froze. I waited 2 hours with no progress. – AJV May 04 '22 at 16:01
  • Can you try using commandline $gsutil cp -m src_url dst_url. This (-m) potion might help us transfer large number of files. Also can you perform perfdiag on your end (gsutil perfdiag -o output.json gs://your-bucket)? If this issue persist please try on Different PC/network/browser/clear browser cache. – JaysonM May 05 '22 at 05:51
  • Both CLI commands in your answer step 5 return 0 when I run them. I have not yet tried the ones in your comment. I have tried this on Chrome with Windows 10 OS. I have also tried this on Firefox with Linux Mint OS. In both cases, browser freezes and upload silently fails. I have tried multiple networks. I have cleared my browser cache. Uploading using a CLI or SDK might work, but it is baffling to me that Google Console can't handle a medium sized file upload without choking out the browser. After trying it on both Windows & Linux on multiple networks & browsers, this may be a GCP issue. – AJV May 10 '22 at 00:06
0

For some reason, uploading the folder along with the files was freezing. I have created a folder, accordingly, and uploaded all the files into the folder. It has worked!

Marlon Teixeira
  • 334
  • 1
  • 14