4

I have 5 web servers (Apache/mod_perl) behind Amazon EC2 Elastic Load Balancing, when I deploy codes to the web servers, I am doing this..

  1. For each machine, shutdown the Apache
  2. Update the code
  3. Start over the server and proceed to the next server

I think when my server is shutdown, ELB will not distribute request to my server, but how about the request still serving?

I think a better approach is

  1. Stop accepting new request from ELB
  2. Sleep for sometimes, shutdown web server only if all requests are responded
  3. Update the codes
  4. Start the server again

But how to perform (1) and (2) from my local sever? Do I need to use AWS API? or other easy way to do it?

Thanks.

Ryan
  • 5,831
  • 24
  • 72
  • 91
  • 1
    1) If you use autoscaling, you can use the API command line tool `as-set-instance-health` to mark an instance temporarily unhealthy. This will give you a short grace period (that associated with the group) before ELB will terminate the instance. If you don't mark it as healthy within that time, the instance will be terminated. 2) Apache has a ['graceful'](http://httpd.apache.org/docs/2.2/stopping.html#gracefulstop) stop, which will allow each child process to finish serving their current requests before terminating. – cyberx86 May 15 '12 at 18:35

3 Answers3

3

I have another way to achieve this, If you have have several things to upgrade on each server. That'll be very wasting to update 5 server for the same operations.

Create a Load Balancing & Scaling Group.Ensure the connection draining is enabled.

There's my upgrade step.

  1. Remove 1 instance from your load balancer. Update everything you need. And create an AMI from you production instance. This will cause the restart of your instance.
  2. Configure the Scaling group with new AMI.
  3. Remove instance from load balancer and the stop the instance. The new instance will start automatically
  4. Continue to shutdown other instance and after all instance re-created. The upgrade is completed.

Some guides:
Load Balance Your Auto Scaling Group
Connection Draining

Happier
  • 131
  • 2
2

I run apache/mod_perl as load-balanced EC2 instances, and do code upgrades regularly just as you say. My process is:

  1. take an instance or two out of rotation
  2. shut down apache on that instance
  3. upgrade the instance(s)
  4. return to rotation, and remove the others

The AWS documentation goes over how to add and remove instance from rotation using either the API or the Console, your choice. You'll notice that with my approach, webservers go out of rotation gracefully, so I'm not worrying about whether a particular user request gets killed. As @cyberx86 mentioned, you can use the command apachectl -k graceful to shut down your apache server after each request is processed.

khoxsey
  • 725
  • 4
  • 9
  • 1
    FYI, when you take the instance out of rotation, it will cut any existing in-flight requests immediately. – mh. Jan 12 '13 at 21:02
  • The outcome of that comment has recently changed with the announcement of connection draining, which will mean open connections can be finished first: http://aws.amazon.com/about-aws/whats-new/2014/03/20/elastic-load-balancing-supports-connection-draining/ – Chris Alexander Apr 10 '14 at 10:52
2

If your deployment can be modified to support newly created instances coming online with new code - you can actually remove the instance from an ELB, wait 60 seconds (after which Amazon's ELB would close the connection to backend and client anyway) and then terminate the instance - and rely on Amazon's Auto Scaling Group to bring a new instance up for you. I've created an open-source tool that automates this process - available at awsmissingtools.com - look for the tool called "AWS-HA-Release."

Colin Johnson
  • 191
  • 1
  • 4