2

I need to backup our Google Storage buckets. Versioning is not enough. I was thinking about:

  • backup to s3 - is there an automated bucket sync from GS->S3 or out-of-the-box solution for scheduled transfers between buckets?
  • backup to another GS bucket - in the same gc project, a coldline bucket "replica" with read-only privs to most users and some automated process to replicate/sync the data?
  • any other ideas?

thanks:)

bartimar
  • 3,374
  • 3
  • 30
  • 51
  • Google cloud does not support any inbuilt service to use for this purpose, so best you can do is run `gsutil rsync` cron job from some machine to trigger this as suggested by @Mike. Also, I would recommend doing the backup on GS bucket if you have a large amount of data, as GS bucket to GS bucket is very fast. – Yogesh Patil Nov 15 '17 at 19:00
  • thanks @YogeshPatil. So I guess the best would be sync it with cron to bucket in different geolocation? Any idea if there is a fee for the traffic between two different coldline/nearline geolocations? – bartimar Nov 15 '17 at 19:20
  • Wait I think this Google cloud transfer service is an answer to your problem. https://cloud.google.com/storage/transfer/. Let me know if it is one which solves your purpose, I will update the answer. :) – Yogesh Patil Nov 15 '17 at 19:25

2 Answers2

3

You could use gsutil rsync to do this:

gsutil -m rsync -rd gs://your-bucket s3://your-bucket

(similarly for syncing between GCS buckets).

You would need to set up a cron job or something similar to cause this to run periodically.

Mike Schwartz
  • 11,511
  • 1
  • 33
  • 36
  • 3
    I'm looking for this "as a service". – bartimar Nov 14 '17 at 15:36
  • I am receiving the following error when trying to do this ` Caught non-retryable exception while listing s3://my-s3-bucket/: AccessDeniedException: 403 InvalidAccessKeyId InvalidAccessKeyIdThe AWS Access Key Id you provided does not exist in our records.{ACCESS_KEY}{REQUEST_ID}{HOST_ID} CommandException: Caught non-retryable exception - aborting rsync ` – 0xC0DED00D Mar 14 '22 at 16:52
  • Posted another question here instead - https://stackoverflow.com/q/71471638/1079901 – 0xC0DED00D Mar 14 '22 at 17:09
3

As mentioned in a comment, GCS Transfer is what you are looking for, at least for the part: "backup to another GS bucket".

From the doc:

Transfer data to your Cloud Storage buckets from Amazon Simple Storage Service (S3), HTTP/HTTPS servers, or other buckets. You can schedule one-time or daily transfers, and you can filter files based on name prefix and when they were changed.

SimonH
  • 248
  • 2
  • 12
  • Hi, would it be enough just to backup to another bucket in gcs or necessary to also backup to aws? – Andrew Irwin Aug 19 '20 at 14:48
  • 1
    It depends on your use case, but I think backup to another bucket is enough, you can also create that other bucket in another region/zone if that feels safer for you. – SimonH Aug 19 '20 at 22:29