1

Goal: Automated full and incremental backups of an AWS EFS filesystem to an S3 bucket.

I have been looking at Duplicity/Duply to accomplish this, and it looks like it could work.I do have one concern, you would have to store API keys in the clear on an AMI for this to work. Is there any way to accomplish this using a role?

flyingcars34
  • 353
  • 4
  • 18

1 Answers1

1

I do backups exactly as you want to and it can be done since duplicity has support for instance profile. Make sure to give appropriate access to your role and attach it to your instance.

deobieta
  • 268
  • 1
  • 8
  • Sorry, I'm still a bit new to AWS. All of the guides I've read through ask you to create a config file with API keys, is this essential to the functionality, or can you skip this if you give the instance a role? Also, may I ask whether you encrypt, and if so do you store the gpg key in the config file? – flyingcars34 Jun 20 '17 at 13:58
  • The idea of IAM roles assumed by an instance is to avoid storing the AWS credentials directly in a config file or being managed by developers, instead the instance with the role can request for temporary credentials to access AWS services. In this case s3cmd takes care of asking and renewing credentials when an instance profile is used by the instance, so you don't have to do that part, just make sure to have s3cmd installed in your instance, give proper permissions to the role with IAM policies and create the config file for duplicity. – deobieta Jun 20 '17 at 15:11
  • I use encryption and I set the GPG key in duplicity config file using ansible vault. – deobieta Jun 20 '17 at 15:14
  • Thank you, that is very helpful. I think I just about have the process figured out, the last time I am trying to figure out is cost/power management. How do you handle keeping the server powered down when it is not needed? I was going to use EC2 Scheduler to power up the machine up shortly before backups are scheduled, but for powering down I would need to know the backups are completed first. Should I build that in to the server itself, and have it scripted to power down when the backup is complete, or is there a better way to do that? – flyingcars34 Jun 21 '17 at 13:07
  • I don't stop the instance because I use it for other services that need to be running all the time. I use [duplicity-backup.sh](https://github.com/zertrin/duplicity-backup.sh) to configure my backups, the wrapper also provides notifications, maybe you can use one of those notifications or add your own hook to the script to shutdown the instance. – deobieta Jun 22 '17 at 19:54
  • I made a mistake on the duplicity information. Duplicity uses boto to use S3 as backend and takes care of credentials when an instance profile is used by the instance. The [wrapper script](https://github.com/zertrin/duplicity-backup.sh) that I use to configure my backups use s3cmd to manage backups. s3cmd also takes care of credentials. Sorry about that. – deobieta Jun 22 '17 at 20:34