1

For an EC2 server I have running I am creating a snapshot of the EBS every night. The EBS has 500GB of space and 1500 / 3000 IOPS. On some days past week the snapshot creating process reached it maximum IOPS causing the server to be unavailable for 5 minutes to an hour. What can I do to prevent the snapshot to use the maximum IOPS? I am creating the snapshot using https://github.com/CaseyLabs/aws-ec2-ebs-automatic-snapshot-bash

Bart Bergmans
  • 633
  • 1
  • 6
  • 9
  • What is running on the server? The performance when taking a snapshot can vary depending on the amount of data, the amount of changed data, and the system load. What about I/O characteristics of your workloads? – dsmsk80 Apr 06 '17 at 08:34
  • Its running a webserver on ubuntu 14.04. The strange thing is that only 3 days past week it reached its max IO. The other days only 5%. – Bart Bergmans Apr 06 '17 at 08:41
  • 1
    I've read a lot about AWS preparing for architect pro, and have the three associate certs. I've never seen any reference anywhere to a snapshot consuming IOPS. It doesn't run from the instance, it's external to your instance, so it shouldn't consume IOPS. All I can find is " EBS snapshots provide enhanced data durability but may slightly degrade your application’s performance while the snapshot is in progress, here https://aws.amazon.com/premiumsupport/knowledge-center/optimize-ebs-provisioned-iops/ . – Tim Apr 06 '17 at 08:42
  • 1
    I ran a snapshot on my instance, CloudWatch didn't even blink, but it's a small disk with not much changed. How much data changes on your disk each day? Try using AWS features to take the snapshot, rather than a third party tool. http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/TakeScheduledSnapshot.html – Tim Apr 06 '17 at 08:43
  • I have now created a CloudWatch schedule to create a snapshot every night. Let's see if this will do anything to the IO. – Bart Bergmans Apr 06 '17 at 09:34
  • 1
    @Tim IOPS and throughput are primarily constrained by the *volume* -- by EBS itself -- not by the *instance*. There's no intrinsic reason why snapshots would *not* consume IOPS -- after all, they are reading data from the volume (directly, not via the instance). For `sc1` and `st1` volume types, this seems like documented behavior: [*"performance may drop as far as the volume's baseline value while the snapshot is in progress."*](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html) – Michael - sqlbot Apr 06 '17 at 11:54
  • @tim EBS snapshots are EBS snapshots, regardless of how they are created. Using third-party tools or your own scripts are equally valid methods as using CloudWatch events. – Matt Houser Apr 06 '17 at 17:25
  • @MattHouser I just wanted to rule it out, I knew it was unlikely. Michael good point and good documentation find. It makes sense that snapshots consume IOPS but in all the documentation I've read, which is a lot, I don't remember it stating it. I doubt there's a way to throttle it. – Tim Apr 06 '17 at 18:48
  • I suggest contacting Amazon about this. First off post in their forums, see if anyone from AWS replies. If not then pay the $29 for a month of support to get the answer. You have little to no control over the snapshot process so there's not much you can do about it. You could use a script to increase the IOPS available before the snapshot, then down again afterwards, but that shouldn't be necessary. – Tim Apr 06 '17 at 19:00

0 Answers0