0

I am trying to download a large dataset from my Google storage in my virtual machine running Ubuntu 18.04 but my download stops halfway. I am using the command:

gsutil cp -r gs://my-storage-bucket/dataset1 /home/user/data

I managed to download a smaller dataset but this one is close to 4 GB in size but every single time it just stops downloading at 1.1 GB. First the MB/S drops and then it stalls at 0.0 MB/S.

I already increased the disk size of my VM but that didn't help either. I removed the file that sits in my folder that makes sure to resume downloading whenever it can, as it doesn't work.

Any suggestions?

brokkoo
  • 157
  • 7

1 Answers1

1

Does it get stall at 0.0MB/s and it does not error out? Can you please try copying it again with the command below?

gsutil \
-o "GSUtil:parallel_thread_count=1" \
-o "GSUtil:sliced_object_download_max_components=8" \
cp gs://my-storage-bucket/dataset1 /home/user/data

Basically, passing two parameters to gsutil to set thread count and max components from sliced object.

Reference: https://cloud.google.com/storage/docs/gsutil/commands/cp

CaioT
  • 1,973
  • 1
  • 11
  • 20
  • Yes, it did not error out. Unfortunately I could no longer SSH into my VM after closing my connection. It got stuck in a loop, which is another issue for which there is no straightforward solution according to StackOverflow posts. I decided to delete my VM and start from scratch because this entire process was taking too long. After doing so and making a new VM, I didn't have the issue. I hope other people can find a fix for this. – brokkoo Jul 28 '21 at 18:32