3

Obviously, Amazon EC2 servers are still running on physical hardware and therefore can have catastrophic failures. And architecturally, I should be building an app that scales horizontally and works around those failures.

How can I simulate an EC2 instance suddenly breaking down? This should have characteristics of a real equipment failure:

  • Processes don't terminate cleanly.
  • Data in memory is not given a chance to write to disk.
  • Files (e.g., on EBS volumes) are not cleanly closed.
  • Open sockets don't FIN, they just hang.
Jeremy Wadhams
  • 877
  • 2
  • 9
  • 19
  • `sudo rm -rf /`? – ceejayoz Apr 22 '13 at 20:26
  • 1
    `sudo kill -9 -1`? – cyberx86 Apr 22 '13 at 20:32
  • 3
    Netflix has [written extensively](http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html) on this, and has [open-sourced](https://github.com/Netflix/SimianArmy/wiki) much of the code they use to simulate failures. – EEAA Apr 22 '13 at 20:35
  • Might be interesting to try: `echo 1 > /proc/sys/kernel/sysrq` followed by `echo b > /proc/sysrq-trigger` (force restart). – cyberx86 Apr 22 '13 at 20:45

1 Answers1

1

There are an enormous number of ways a system can fail so you probably can't test for all of them and try to work around them.

Perhaps you should look at it from another perspective: look for the services which are essential then find a way to explicitly kill these at random. This simulates a failure you care about, regardless of the cause.

For example if your instance runs an httpd and an FTP server you can kill these daemons occasionally and make sure you can recover from it. You can even programatically terminate the whole server with the AWS API if you want to.

This will also exercise your monitoring infrastructure if the recovery doesn't work :-)

ianjs
  • 410
  • 2
  • 6