2

I have a problem with autoscaling with AWS Application Load Balancer.

I'm running my Jmeter tests and discovered that whenever new instance is added to autoscaling group (that is when it becomes healthy and ALB starts to route traffic to it), then for some short period of time Load Balancer forwards less requests to targets and a lot of requests are apparently stuck at Load Balancer itself.

I'm attaching 3 images that show this issue. CPU of JVM of one of instances drops and than goes back to normal, some requests are hanging for more than 30 sec. Number of requests per target drops and then goes back to trend. (see attached pictures)

I'm using sticky sessions with 3 minutes validity period.

Does any one knows what may cause this temporary "choking" when new instance is added?

It is quite crucial to our user experience. Can't actually understand why adding new instance can have such adverse effect on traffic routing.

Issue is fully reproducible.

Max response time grows rapidly when new instance is added

Requests count per target drops when new instance is added

CPU consumption drops when new instance is added

Marcin Tomiak
  • 331
  • 2
  • 8
  • It looks like the new instance isn't warmed up, so it's serving requests more slowly than the existing nodes, thereby increasing average request duration, causing fewer requests to be attempted until its performance is on par with the others. – Michael - sqlbot Apr 06 '18 at 00:34
  • Plot of CPU that you see (from JVisualVM), represents instance that is warmed up. As I wrote in a question, I'm using 3 minutes sticky sessions, so why when new instance is added, CPU on warmed up instance drops so much? It looks like Load Balancer does not send as much traffic as it used to. And then all goes back to trend. – Marcin Tomiak Apr 06 '18 at 06:53

0 Answers0