0

tldr; How do you configure the Apache MPM event module as a functional reverse proxy when running behind a high throughput AWS Application Load Balancer?

Layout

AWS Application Load Balancer -> Apache 2.4.x -> AWS Internal TCP Load Balancer (NLB) -> Jetty App Servers

Using the configuration setting shown below the Apache MPM event module is not stable behind an AWS app load balancer. Response times are greatly increased and requests intermittently fail for 502 errors returned by the ALB.

Worker configuration

ServerLimit 250
StartServers 100
MinSpareThreads 75
MaxSpareThreads 250
ThreadLimit 64
ThreadsPerChild 64
MaxRequestWorkers 8000

ProxyPass config

enablereuse=on acquire=10 connectiontimeout=2 timeout=55 max=400 retry=0 keepalive=on

MPM event vs MPM worker

Switching to the MPM worker module shows instant improvements in response times and the 502 errors disappear completely.

Target Response Times - MPM event to MPM worker

502 Errors - MPM event to MPM worker

Debugging

I managed to trace the 502 errors returned by the ALB to 408 errors returned by Jetty. Jetty seems to be throwing the errors because of timeouts reading the request.

Running Ubuntu 20.04 with Apache 2.4.41-4ubuntu3.8.

Interesting notes

  • Requests though an AWS Network Load Balancer (NLB) to the same Apache instances respond quickly and don't show any 502 errors
danw
  • 101
  • 2
  • The difference with the NLB may be because NLBs work at a lower layer than ALBs. Can I ask why apache is in the middle? Why not just have the ALB route to the Jetty servers? – shearn89 Dec 21 '21 at 15:57
  • @shearn89 its a WAF: https://github.com/SpiderLabs/ModSecurity – danw Dec 23 '21 at 20:44
  • Why not use AWS WAF instead, which is probably cheaper than running a whole EC2 instance? plus it's managed for you, so you just have to enable some AWS-managed rules and you're good to go! – shearn89 Dec 24 '21 at 08:46

0 Answers0