I’m load testing a NestJS application running on ec2 in AWS. If I configure my load testing tool to hit the NestJS application directly, I can hit it with thousands of requests concurrently without an issue.
However, if I execute the same load test but through an Application Load Balancer (ALB), I start to receive frequent 502s – even with as low as 50 concurrent requests.
I’m using the Express framework and it looks like many people have had a similar issue with using Express behind a load balancer. The usual recommendation is to set the keepAliveTimeout and headersTimeout to values that are larger than the load balancer’s idle timeout, which I’ve tried doing in a few different ways that haven't solved the problem:
const server = await app.listen(config.server_port);
server.keepAliveTimeout = 120 * 1000;
server.headersTimeout = 125 * 1000;and..
const adapterHost = app.get(HttpAdapterHost);
const httpAdapter = adapterHost.httpAdapter;
const express = httpAdapter.getHttpServer();
express.keepAliveTimeout = 120 * 1000;
express.headersTimeout = 125 * 1000;It appears that Node is emitting this event: https://nodejs.org/dist/latest-v8.x/docs/api/http.html#http_event_aborted which is resulting in this stack trace from the Express ErrorHandler:
npm: BadRequestError: request aborted
npm: at IncomingMessage.onAborted (/var/lib/hdai-apps/compute-api/node_modules/raw-body/index.js:231:10)
npm: at IncomingMessage.emit (events.js:315:20)
npm: at IncomingMessage.EventEmitter.emit (domain.js:467:12)
npm: at abortIncoming (_http_server.js:561:9)
npm: at socketOnClose (_http_server.js:554:3)
npm: at TLSSocket.emit (events.js:327:22)
npm: at TLSSocket.EventEmitter.emit (domain.js:467:12)
npm: at net.js:673:12
npm: at Socket.done (_tls_wrap.js:563:7)
npm: at Object.onceWrapper (events.js:422:26)
npm: at Socket.emit (events.js:315:20)
npm: at Socket.EventEmitter.emit (domain.js:467:12)
npm: at TCP.<anonymous> (net.js:673:12)
I've created an Express error handler and can capture a little more information:
{"message":"request aborted","code":"ECONNABORTED","expected":35165,"length":35165,"received":8948,"type":"request.aborted"}
So Node is basically telling me that the client disconnected, but the ALB logs are telling me that the target disconnected.
Not sure where to take this next. Would greatly appreciate any advice from somebody who has successfully configured NestJS/Express to run at high load behind a load balancer. Thanks!
