I have a node/express app in a Fargate container, which serves static files (html/css/js) in addition to hosting an API. The network components in front of it look like so:
(WAN) ---> (HTTPS HTTP/2) Proxy ELB ---> (HTTP HTTP/1) nginx reverse proxy ---> (HTTPS HTTP/1) App ELB ---> Fargate container
That might not be a standard stack, but it is HIPPA compliant, which is necessary for our application.
At some point in this chain, a connection is being dropped intermittently while responding to requests for the statically served files. It seems to happen to the same file repeatedly (say the main.js file, but has happened to a 377 byte icon) and happens about once every ten times I attempt to load the site. In Chrome DevTools, I can see that the request is pending for sixty seconds, after which I get an HTTP 504 back from one of the ELBs in the chain. I can change the timeouts of the ELBs and get a HTTP 502 instead because the ELB isn't timing out.
I've been all over the place trying to chase this down, including fiddling with the app itself (upgrade express, peek under the covers as much as possible to see when connections/sockets open and close) and I'm convinced that the application itself is not the problem (also, I'm hosting the app in 3 environments; this problem happens in 2 of them. Additionally, other node/express apps we've deployed with the same network stack work fine). I've come to be convinced the problem is with the AWS ELBs, nginx, or some complicated interaction between them.
nginx.conf (relevant portions):
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Authorization $http_authorization;
proxy_pass_header Authorization;
gzip on;
gzip_types text/plain application/xml application/javascript;
gzip_proxied no-cache no-store private expired auth;
gzip_min_length 1000;
if ( $http_x_forwarded_proto != 'https' ) {
return 301 https://<site URL>$request_uri;
}
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
location / {
proxy_pass https://<app ELB>;
}
Any help is greatly appreciated!
