0

We are using EC2 spot instances for crawling data. At any given time we need to have 20 instances running. We are aiming for high availability as well as cost optimization.

The scheme we want to follow is below: We start 20 spot instances with a very high bid ($5), in order to avoid termination. We keep monitoring the spot price and when we find that the spot price is above $2 and remains there for more than N hours, we launch 20 on demand instances and then terminate all the spot instances.

Again keep monitoring the spot price and when spot price settles down, launch 20 spot instances and terminate the on demand instances.

To keep things simple we are looking for an off the shelf tool to handle this for us. Can I do this using autoscaling or any other tool?

Varun
  • 405
  • 2
  • 4
  • 7

1 Answers1

1

I'm not aware of any tool that will do this for you. You can use auto scaling for the spot instances but its not going to do what you're describing here except that its an easy way to control how many instances are currently running.

We tried a similar setup before where I work and we thought the same thing. Set up a high bid and we will never have the instances destroyed. Let me tell you that even with high bids you will get terminated instances and if you're very unlucky you will actually have to pay that very high bid. About a year ago there were lots of people doing this, people who would put crazy bids upwards of $100 for some instances and wound up having to pay for them.

There are plenty of examples if you look at the history of the prices. Looking at the c1.medium in us-east-1a you can see that the price jumped from $0.018 to $.165 on Jan 24th. This would effectively terminate all your instances running without any notice. I will warn you that this setup is risky.

However, there are significant savings you can achieve with this type of model, there are successful companies who are able to pull it off but its not easy and there's no simple tool (that I know of) that will do it for you. It really depends on your use case. If you need 100% up time this is not a good solution unless you have some kind of hybrid system where you always have on-demand instances running in case there is some kind of failure in the spot instances.

We decided to scratch the whole thing after it just became too much of a pain to maintain and achieve 100% up time. For us the cheapest solution was to use reserved instances, they are much cheaper than the on-demand instances and with the new pricing model for the reserved instances i'm sure you could find something that fits your use case now that you can have Light Medium or Heavy Reserved instances.

bwight
  • 791
  • 1
  • 6
  • 14