0

I have an autoscaling group in AWS EC2, on which I have an autoscale policy, which keeps the loadbalancer Request Count Per Target at 800 and it has a warm up time of 250 seconds.

Now, every time a new machine is booted up, regardless of the warm up time, that machine is receiving requests. Since the application inside the machine is not up and running, those requests are failing.

Is there anything else I have to do to avoid this behavior?

Sebin
  • 101
  • 2
  • When an ELB believes that *all* targets are unhealthy, it assumes you have a problem with your health check configurarion or behavior and fails safe -- sending requests to all targets (instead of rejecting all requests). Have you that your active instances are indeed passing their health checks with the balancer? – Michael - sqlbot Mar 22 '18 at 09:04
  • @Michael-sqlbot In my case, rest of my machines are healthy. In fact, the newly added machine could also show a healthy status even in the warm up duration. – Sebin Mar 22 '18 at 09:37
  • A couple of points. It sounds like the warm up timer may not be what you assume -- as far as I am aware, that's strictly a mechanism that prevents auto-scaling from overreacting to workload changes by assuming its scale-out actions weren't sufficient and that additional rounds of capacity increase are needed, giving you time to stabilize after scale-out. As soon as the instance passes its ELB health check (confirm: we are not talking about EV2 system or reachability instance health checks, here -- it's ELB health checks), it should start receiving traffic, but not before then. Thoughts? – Michael - sqlbot Mar 22 '18 at 10:03
  • @Michael-sqlbot The mechanism that prevents further autoscaling is called cooldown period. Like you said, it could be because of the health check passing before the app is running, the machine is receiving requests. – Sebin Mar 22 '18 at 10:40
  • You are correct... I did misstate what [warm up](https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-scaling-simple-step.html#as-step-scaling-warmup) is -- it makes *other* changes to the way auto-scaling reacts while the instance is still within the time window allocated for picking up its share of the load (e.g., "hmmm, I seem to need 3 more instances... but I have 1 still in its warmup window, so I'll only add 2 more for the moment")... but it doesn't prevent the instance from taking traffic. You definitely need to deliberately fail your ELB health checks until you are indeed healthy. – Michael - sqlbot Mar 22 '18 at 12:36
  • @Michael-sqlbot I'll try and post here. Thank you for helping out. – Sebin Mar 22 '18 at 13:29

1 Answers1

0

There was an issue with my setup, I had set up Nginx in such a way that health_check doesn't reach rails application. So even before the application is loaded, health check was passing. That's why the new machines were receiving requests.

Now, I changed the config to hit the rails application. Now until the application is booted, health check will fail, so that no requests will be routed to new machines.

Thanks @Michael-sqlbot for helping out.

Sebin
  • 101
  • 2