2

I read this question and answers: Will my EC2 Spot Instance Volumes Die with the Instance?

As a result, when I requested a Spot Instance I added a second volume under EBS volumes (the standard volume it gave me) and unchecked the Delete tag.

I then SSH launched my instance once it was ready and installed what I needed to install (Tensorflow, iPython, Jupyter, etc...packages for Machine Learning).

Then I launched a jupyter notebook with jupyter notebook --certfile=~/certs/mycert.pem --keyfile ~/certs/mycert.key, created a new notebook, saved it and terminated the Spot Instance to test whether all of those installations and my notebook would stay on the second non-delete volume I'd created with my spot instance.

When I went to terminate, it did confirm that the second EBS volume would NOT be erased.

I then launched another Spot instance without specifying a second EBS volume this time, and all of my installs and the notebook were gone, when I tried to launch jupyter I got a "command not found" error.

Do I need to somehow "cd" into that specific volume after I SSH launch the instance? Isn't everything I do automatically saved on that second EBS volume? Do I need to take a Snapshot, or create an image of the second EBS volume?

What's the easiest way to keep my installs and files no matter how suddenly my Instance terminates and access them at a later point from any other instance? I looked at the EBS pricing and don't mind paying the $0.1/GB/month, I'm just not sure how to make it work.

Community
  • 1
jeremy radcliff
  • 123
  • 4

2 Answers2

6

In general, you need to ensure that any state you care about is stored on the secondary EBS volume.

I will say, though, that you're doing it wrong if you are manually installing all of this on a spot instance. By doing so, you're setting yourself up for failure, and for an absurd amount of boring repetitive tasks to slog through on your way to that failure state.

Instead of doing what you're doing, fire up an on-demand instance and get it configured how you want with regards to software installs, credentials, services, etc. Then make an AMI of it. With this image, you can then create spot instances that are 100% ready to go from initial boot.

With regards to retaining other state, you'll either need to re-work your workflow to keep state on S3 or else mount a non-deleting EBS volume to a certain location on the filesystem and then ensure that all of your state is stored on that filesystem so that it can be attached and re-mounted to a future spot instance.

Michael Hampton
  • 244,070
  • 43
  • 506
  • 972
EEAA
  • 109,363
  • 18
  • 175
  • 245
  • When you say "with regards to retaining other state", do you mean for example saving a specific iPython notebook? Would that be lost with the image? Is the image really just for software installs like iPython, Jupyter, git, etc...? – jeremy radcliff Oct 28 '16 at 00:21
  • Yes, that's correct. "State" refers to anything you want to keep around when the instance dies **that isn't installed** already in the image you start the server from. So yes, iPython notebooks and other misc files are included. – EEAA Oct 28 '16 at 00:22
  • That makes sense. Thanks a lot for your help, this stuff is confusing at first. – jeremy radcliff Oct 28 '16 at 00:24
0

With improvements from AWS on spot instances you now have the ability to stop or hibernate the instances too instead of getting terminated. Suggestion on building the AMI in the accepted answer is still relevant and must be done as a best practice.

Prabhat
  • 183
  • 5