3

When the autoscaling rules in my group trigger the deployment of a new EC2 instance, or for example I change the instance size to simulate the behaviour the node is deployed but it is added to the load balancer before the application is ready.

The deployment starts and the scripts in .ebextensions begin to execute. The folder entitled current is empty as it's the first deployment to this node so accessing this node displays a 403 Forbidden error in the browser.

Eventually the application deployment finishes and the ondeck folder is moved into current as expected, at which point the application works and the 403 error disappears.

The problem here is that the node is added to the load balancer before this process completes so some of the traffic is distributed to broken nodes.

Is there any way I can stop this happening? Am I doing something wrong?

Ben Swinburne
  • 337
  • 4
  • 17

1 Answers1

6

The load balancer will start sending traffic towards the new EC2 instance as soon as that instance starts passing the load balancer's health check. I'm guessing your load balancer is configured with a TCP ping health check. That means as soon as the new instance starts responding to pings on port 80 it is considered healthy and ready to receive requests. Like you've noticed, though, that will happen before the app is fully deployed and actually serving up good responses.

The solution is to configure your load balancer to perform a health check with HTTP requests, so that the new EC2 instance will not be considered healthy until your app is running and successfully responding to web requests. On the AWS console find the Load Balancers section of the EC2 dashboard and select the load balancer for your Elastic Beanstalk environment. You'll see a Health Check tab in the load balancers configuration panel where you can edit the health check settings and switch to HTTP requests as your ping method.

If your application is configured to only handle requests sent via the production hostname, you may find that the HTTP health check always fails. The load balancer sends requests to the specific URL of the EC2 instance, and if your application responds to those requests with errors or 30x redirects, the load balancer will consider it unhealthy. In that case you'll need to add an exception to your request filtering to allow the health check request to succeed regardless of hostname. For example, add a static, empty elbhealthcheck.html (or whatever you want to name it) file to your application and allow requests for that file with any hostname. Then configure the load balancer health check to use that URL in its HTTP ping, and you'll be good to go.

Nick Park
  • 61
  • 1