1

I'm baking an AMI from a database server which has 300GB root volume. 80% of the volume is in use. Reason behind baking the AMI is that we need multiple new instances with the exact same data everyday. AMI is the appropriate solution because the restoration process is extremely slow. So the data restoration process can't be initiated after creating the instances. We want instances to be ready in 7-8 minutes with all the data.

But, the performance in the new instances is extremely poor. The reason behind it is the instances use EBS and that needs to be initialized as described in this doc.

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html

Unfortunately, the initialization process takes 5-6 hours and which is not a solution for us.

So, what is the best practice to bake an AMI when the underlying data needs to be in AMI is really big?

Ashish Bista
  • 4,533
  • 1
  • 18
  • 25
  • What database engine are you running on these instances? – Michael - sqlbot Sep 18 '17 at 19:35
  • @Michael-sqlbot It's not related to database engine. We are not using MySQL or PostgreSQL. The restoration is a custom process. – Ashish Bista Sep 18 '17 at 21:46
  • I understand it isn't related -- this is part of the nature of AMIs and EBS volumes from snapshots, regardless of what specifically you're running. Depending on what you're doing, Aurora copy-on-write clones or EFS might be viable solutions, but in any event it should be possible to warm up a 300 GB volume in less than 5-6 hours. – Michael - sqlbot Sep 18 '17 at 22:40
  • Can you sync them to S3 while your EBS is running? And when you create new instance, use user-data to sync from S3->EBS. Use VPC Endpoint with max speed and minimum price. Another way, how about EBS Snapshot incremental? – Bui Anh Tuan Sep 19 '17 at 00:35
  • @TuanBA Our instances go in production as soon as they got created. Seems like this is not possible as of now. – Ashish Bista Sep 19 '17 at 17:42

2 Answers2

5

Now, I have something that helped a lot in initializing an EBS volume.

AWS recommends dd or fio for initializing EBS volumes. Running a single dd process takes too much time. So, having multiple processes of dd to pull a small chunk of data from given block makes the initialization process really quick.

nohup seq 0 $(($(cat /sys/block/xvda/size) / (1 << 10))) | xargs -n1 -P8 -I {} sudo dd if=/dev/xvda of=/dev/null skip={}k count=1 bs=512 > /dev/null 2>&1 &"

Ashish Bista
  • 4,533
  • 1
  • 18
  • 25
  • 2
    would you mind sharing how long this above process takes ? (compared to the 5-6 hours that you mentioned above) – shluvme Apr 23 '20 at 13:38
  • 2
    Consider using EC2 Warm Pools in conjunction with an EBS Warming Technique to further improve disk performance of ephemeral instances created from an AMI. AWS also offer a "Fast Snapshot Restore" feature but last I checked it was quite costly. – Ryan Williams Jan 02 '22 at 05:09
  • @shleimel I don't remember exactly but it was like 80-90% improvement. However, as suggested by Ryan, consider using EC2 Warm Pools. – Ashish Bista Jan 25 '22 at 01:51
  • Thanks @AshishBista, I had a similar problem but with a host hypervisor loading guest VMs. Ultimately I ended up pulling the guest VM images from an S3 bucket (and not baking them into the AMI) in order to have the "hot" on the EC2. – shluvme Jan 26 '22 at 06:40
  • 2
    @shleimel I had good results with the following undocumented approach. You can launch the AMI with io2 10k iops, and then immediately modify the ebs to regular gp2 or gp3 volume. In my experience, if the drive is under 50GB, you will get most of the blocks initialized already. – jellycsc Jan 26 '22 at 16:19
  • @jellycsc That's a smart idea. I didn't know that ebs types can be converted after creating them. Thanks! – Ashish Bista Jan 28 '22 at 03:04
2

Amazon is now providing a service call Fast Restore Snapshot. Using this you can mark the Snapshot of your AMI as a Fast Restore Snapshot for the region/availability zone you need to create new instance. It may take up 1hr for 1TiB of volume and is chargeable.

However, the benefit here is that the volumes created from this snapshot are ready to use at full capacity of the iops provisioned. This way, you wont have to wait or do any manual reading of the blocks.

You can disable the fast restore once you have created the volume to prevent any further charges from accumalation.

Ketan Patel
  • 157
  • 1
  • 2
  • 6