1

We are currently evaluating using Google Storage Nearline for backups. The servers in question are located within a Hetzner data center, on average 80-100Mbit/s upstream and 300-500Mbit/s downstream. We're using Googles gsutil for accessing the storage and its rsync command to synchronize our backup directories. While downloads are adequately fast, uploading files takes way too long: Maximum upload speed is about 8-11Mbit/s. This is only happening with gsutil and not really usable for a backup storage location. I know large files should be uploaded using parallel composite uploads, however Google recommends to not enable it for Nearline.

Now, my question is, are these low upload speeds expected? Is there a way to optimize it? As it is, we generate more data than we can push to Google Storage, and will have to look for other solutions.

Moritz Friedrich
  • 185
  • 1
  • 9

1 Answers1

1

If you have a large number of objects to synchronize you might want to use the gsutil -m option, to perform parallel (multi-threaded/multi-processing) synchronization: gsutil -m rsync -d -r data gs://mybucket/data

“BE CAREFUL WHEN USING -d OPTION!”.

Kamran
  • 1,425
  • 7
  • 17