1

I need to backup my aws s3 bucket with lots of files (800k files, 400GB) to google cloud storage. Google Storage Transfer Service seems like an easy solution but i have some doubts about cost-efficiency.

First time it will download a whole s3 bucket:

  • 400 GB outgoing traffic on aws
  • 800k get requests on aws
  • 800k put requests on google storage

What about the second time and later?

It will cost the same as first time or it will download only changed/new files?

Ash Ahluwalia
  • 43
  • 4
Sergey Babochkin
  • 173
  • 3
  • 7

2 Answers2

1

Per the documentation at Creating and Managing Data Transfers with the Console: "By default, Storage Transfer Service only overwrites an object when the source version is different from the sink version."

Therefore, the cost of the second time will be the cost of the list operations on the S3 and GCS buckets (needed to compare to see what changed), plus GETs and PUTs for whatever changed. I believe objects are typically listed 1,000 at a time (according to the maxmimum page size for object listings in each service). Therefore, you could expect the cost in your scenario would be approximately 800 list operations in S3 and 800 list operations in GCS for each subsequent transfer if no objects changed.

Travis Hobrla
  • 5,411
  • 23
  • 22
0

Ingress to gcp is free. So, create a new backup every time, and keep only latest 5 (or whatever). This way you don't need to worry about overwrites etc, plus you can rollback any file up to N versions ago.

Oliver
  • 27,510
  • 9
  • 72
  • 103