2

I know that we can bid on spot instances and get them at lower prices than that of regular instances, but with spot instances there is the risk of your instances being taken back.

I want to know that is there any way we can ensure that they are taken away only when our jobs are finished?

j0k
  • 22,600
  • 28
  • 79
  • 90

1 Answers1

5

The only way to 'ensure' it would be to bid a very very very very high value!

But realistically, as you know, if you request spot instances, keep in mind that if the current spot price exceeds your max bid, either instances will not be provisioned or will be removed from the current job flow. To hedge the complete lose of a jobflow, multiple instance groups can be created where the CORE group is a smaller complement of traditional on-demand systems and the TASK group is the group of spot instances. In this configuration, the TASK group will only benefit the mapper phases of a job flow as work from the TASK group is "hand back up" to the CORE group for reduction.

So say if you have to run a job which would ideally need 40 slave machines, then you can have say 10 machines(CORE group) as the traditional instance while other 30 as spot instances(TASK group). The syntax for creating the multiple instance groups is below:

elastic-mapreduce --create --alive --plain-output
...
--instance-group master --instance-type m1.small --instance-count 1 \
--instance-group core   --instance-type m1.small --instance-count 10 \
--instance-group task   --instance-type m1.small --instance-count 30 --bid-price 0.018
Amar
  • 11,930
  • 5
  • 50
  • 73
  • Ok, I do not get it, what do you mean by having two groups? Can we really do this? I mean can you give an example command? –  Dec 23 '12 at 21:19
  • Updated the answer to include more details along with the cli example. – Amar Dec 23 '12 at 21:25
  • Ah got it man. That's pure awesomeness... this way even though all the spot instances are taken away still we have the core group intact. Brillaint :) thanks again. –  Dec 23 '12 at 21:26
  • 4
    Yeah we do use it regularly, but sometimes a half hour job might take more than an hour if your bidding is outbid too frequently and you are using them on a very busy day! – Amar Dec 23 '12 at 21:28
  • Ya got that, will take care of it :) –  Dec 23 '12 at 21:28