Is there software that can help me easily back up my data from Amazon S3 to my premise?

Question

So everyone backs their stuff up to the cloud. But I want to back up my stuff from the cloud.

For a planned SaaS service, users are saving files containing their intellectual property and we're saving them in S3. I'm not entirely comfortable with the only copy of a file they are entrusting to me being at a third party. So I'd like to do a good encrypted, deduping, etc. backup from S3 to my on premise (or even from AWS straight to Iron Mountain or something). Anyone have an idea for this (besides "write scripts," yes, I could do that, but I'm happier to use software somewhat meant for the purpose).

The thing that seems to disqualify most normal backup software is that it would need to consume Amazon S3 as a source (not a destination - I know a lot of stuff does it as the destination). I'd want it to encrypt the files and then put them somewhere and ideally not transfer files that haven't changed. I'm actually not all that picky about other details. It can be Linux or Windows based, I can run it on an instance in the cloud or on a server on my premises, it can use whatever transit protocol.

I imagine if there's a good production-ready "mount s3 as a drive" option, then really many backup programs could be brought to bear including "ssh+rsync". I see a couple of these but they all look sketchy.

Is anyone doing this or know how it might be most easily accomplished? Open source is good but I'm also willing to toss some money at it if it gets me nice features.

You're on the right track, multiple copies is the only way to be sure your data is safe. — Nate, Oct 13 '10 at 21:40
@Nate - in this case, multiple copies isn't what the OP is concerned about. S3 by default, stores multiple copies of each object in multiple datacenters. What Ernst is concerned about is the "all-your-eggs-in-one-basket" (that basket being Amazon AWS) issue, which IMO is a very valid concern. — EEAA, Oct 13 '10 at 21:45
@ErikA My point is that it is only as good as one copy since its all under Amazon S3. Even if they have "redundancy" its not an extra copy... — Nate, Oct 13 '10 at 22:06

score 2 · Accepted Answer · answered Oct 14 '10 at 03:49

2

The only thing I've seen that even comes close is s3cmd's sync command; it's bi-directional, which is what you want.

answered Oct 14 '10 at 03:49

pjz

10,595
1
32
40

I'm gonna use this for the time being; it's a shame more full featured backup software doesn't consume s3 yet, seems like it's an easy enough addition! – Ernest Mueller Oct 18 '10 at 15:02
The other answer is the one you didn't want to hear: "write scripts" :) – pjz Oct 19 '10 at 02:40
How is this different from the s3 sync command that comes with the official aws cli? – Olivier Lalonde Mar 20 '16 at 22:24
...it predates it! – pjz Mar 24 '16 at 18:22

score 0 · Answer 2 · answered Oct 13 '10 at 23:17

0

If you plan to do manual backups try http://www.s3fox.net/

It works well for us.

answered Oct 13 '10 at 23:17

FrankJK

306
1
3

Thanks; I'm looking for something automated/automatable. – Ernest Mueller Oct 14 '10 at 03:13

Is there software that can help me easily back up my data from Amazon S3 to my premise?

2 Answers2