0

I've read through the Amazon doc, and searched through countless of posts in SO but still I've not been able to answer 2 questions.

Situation: As I understand, Amazon RDS snapshots are incremental. But how can manual snapshots play nice with automated snapshots feature ? Imagine I have created a script to snapshot the DB every day at 1 A.M. and copy the snapshot to another region. Then the snapshots will be:

+ Day 1 - 1 A.M.: snapshot 1 (manual) => copy to the other region (first time, full copy)
+ Day 1 - 6 A.M: snapshot 2 (automatic backup), in current region, not copy.
+ Day 2 - 1 A.M.: snapshot 3 (manual) => copy to the other region (incremental)
+ Day 2 - 6 A.M: snapshot 4 (automatic backup), in current region, not copy (incremental)

Question 1:

snapshot 3 = changes since snapshot 1 or snapshot 3 = changes since snapshot 2 ?

Question 2:

If I just copy snapshot 1 and snapshot 3 (manual snapshots) to another region, is it enough to restore the database ?

In storage world, snapshot 3 = changes since snapshot 2, and we must have all the incremental snapshots (1+2+3+4) to reconstruct the original volume. But for AWS, there're implications and I'm not so sure.

Thank you.

Tung Vs
  • 113
  • 10

1 Answers1

0

Question 1: snapshot 3 = changes since snapshot 2

Question 2: If you copy a snapshot to another region, the first copy will become a full snapshot and not an incremental and yes you would be able to restore the database from a copied snapshot. Subsequent snapshot copies will be incremental based upon the last copied snapshot.

Once a DB Snapshot is copied to a specific AWS Region, any subsequent copy operations of the DB Snapshots of the same DB Instance to that Region will only transfer the data that has changed since the last copy. Thus the subsequent copy operations transfer fewer amounts of data and complete faster.

copy of Amazon RDS DB Snapshots across AWS Regions

John Hanley
  • 74,467
  • 6
  • 95
  • 159
  • Thank for quick response. Does your answer mean to restore the DB to snapshot 3, we need all the snapshot 1, 2 and 3 ? So I have to copy all the snapshots, manual **AND** automatic, to the other region ? – Tung Vs Nov 14 '17 at 03:47
  • If you are restoring snapshot 3 in the region that you created it, then you need all parent snapshots. If you copy snapshot 3 to another region, then you only need snapshot 3 as snapshot 3 will become a full copy if it is the first snapshot copied. – John Hanley Nov 14 '17 at 03:49
  • I will test it right after my AWS account is unblocked. There are so many complications in billing with Amazon than with other companies. Thank you for you info, Mr Hanley. – Tung Vs Nov 14 '17 at 04:32
  • And lots of features so that you can control your costs. Everything is 'a la carte'. Very quickly everything will seem easy. – John Hanley Nov 14 '17 at 04:39
  • *"If you are restoring snapshot 3 in the region that you created it, then you need all parent snapshots"* No, you don't. If you delete all but one snapshot, any snapshot, you can still restore the entire database as it existed when that one remaining snapshot was made. Snapshots don't contain data. They contain lists of pointers to backed-up blocks of data, and those blocks are not purged from backup storage until their reference count reaches 0 (no snapshot references them). Because of this mechanism, the unchanged blocks were already copied, and subsequent transfers can be avoided. – Michael - sqlbot Nov 14 '17 at 04:48
  • Michael, I will defer to you on this one. However, Amazon documentation uses this wording "In State 2, the volume still contains 10 GiB of data, but 4 GiB have changed. Snap B needs to copy and store only the 4 GiB that changed after Snap A was taken. The other 6 GiB of unchanged data, which are already copied and stored in Snap A". As an engineer who has designed snapshot file systems, when there are multiple snapshots, there can be multiple copies of the same data location with different values from different points in time. These different point in time copies must be stored somewhere. – John Hanley Nov 14 '17 at 04:55
  • [*"Amazon RDS creates a storage volume snapshot of your DB instance, backing up the entire DB instance and not just individual databases."*](http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithAutomatedBackups.html). A "storage volume snapshot" -- in other words, it's really an EBS snapshot (except for Aurora). – Michael - sqlbot Nov 14 '17 at 04:57
  • Interesting, then "an entire DB instance" would not be an incremental snapshot but a full snapshot. My answer is based on incremental snapshots. – John Hanley Nov 14 '17 at 05:00
  • No, it *is* incremental. It's incremental at a storage block level because it's a snapshot of the underlying disk, and those are inherently incremental (in EBS). – Michael - sqlbot Nov 14 '17 at 05:02
  • On the page you cited, they've oversimplified it for illustration purposes. On that same page, *"Even though snapshots are saved incrementally, the snapshot deletion process is designed so that you need to retain only the most recent snapshot in order to restore the volume."* – Michael - sqlbot Nov 14 '17 at 05:06
  • Here's a the clencher: *"An EBS snapshot contains a set of pointers to blocks of data, including the blocks shared between multiple snapshots. Each time a new snapshot is taken of an EBS volume, only the data that has been modified since the last snapshot is pushed to S3. When a snapshot is deleted, only the blocks not referenced by later snapshots should be deleted."* -- https://aws.amazon.com/message/2329B7/ (The entire document is interesting reading.) – Michael - sqlbot Nov 14 '17 at 05:07
  • This means that there is a separate EBS volume backing each snapshot. You cannot use one EBS volume. Let's say file system block 1000 is modified in between each snapshot. You will need a unique copy of block 1000 for each snapshot. This would require separate EBS volumes or linked volumes. If the EBS volumes are "sparse volumes", then you only store the changed blocks. Therefore, you need the previous EBS volumes for incremental blocks. There is no magic that Amazon can apply to get around this. *** I did not see your last two message when I wrote this. – John Hanley Nov 14 '17 at 05:07
  • Michael, let's take this to private email. You and I both have access to vendor information and we are very advanced in this area. I would really like to make sure that my detailed understanding of snapshots is correct in regards to AWS. I may be making assumptions based upon internals of NTFS, NSS and Linux. – John Hanley Nov 14 '17 at 05:12