0

I have a website architecture as follows:

internet --> loadbalancer --> webserver/api

So there is an nginx on the load balancer machine setup as a load balancer and there is also an nginx on the webserver/api node functioning as a reverse proxy. The webserver server receives requests from browsers (via the load balancer), accesses the api over HTTP and renders the page to the browser. The webserver and api are both nodejs apps.

The nginx load balancer has log entries for the webserver-->api connections, but it doesn't log the initial client browser-->webserver connections until the browser is closed (tested with Chrome and Firefox). It's as though the connection is kept in an unfinished state until the browser is fully shutdown, at which point the log entry is written.

nginx load balancer access logs:

110.110.110.101 - - [21/Feb/2019:22:21:23 +0000] loadbalancer01 TCP 200 186833 825 0.047 upstream: 10.0.0.100:443
110.110.110.100 - - [21/Feb/2019:22:21:37 +0000] loadbalancer01 TCP 200 24327 3856 21.991 upstream: 10.0.0.100:443 <-- only created after browser is closed
  • 110.110.110.100 - ip of client connecting with Chrome/Firefox
  • 110.110.110.101 - webserver/api node public interface
  • 10.0.0.100 - webserver/api node private interface

The webserver->api connection is logged first even though it clearly happens second, and the client browser->webserver connection only gets logged when the client browser is completely closed.

Is there some sort of buffering happening? I'm not using the buffer parameter in the stream block logging configuration:

log_format combined '$remote_addr - - [$time_local] $hostname $protocol $status $bytes_sent $bytes_received $session_time upstream: $upstream_addr';

access_log /var/log/nginx/access.log combined;

Why does the connection only get logged when the browser is closed? How can I ensure that the initial connection is logged when the connection happens?

[update - added log configuration, also note that ips have been redacted]

vy218
  • 707
  • 1
  • 8
  • 15

1 Answers1

0

I figured this out by comparing the headers between a browser connection to the load balancer compared to a connection initiated from a script. Turns out the browsers set "Connection: keep-alive" header which keeps the connection open so multiple requests can be sent using the same connection.

Useful commands to run this on the load balancer public ip to see the connection headers:

sudo tcpdump -nn -A -s1500 -l -i eth0 port 80

The other thing to note is that if you are using ufw as firewall, it sets up the underlying iptables rules with limits so it only logs the 1st 3 connections per min.

vy218
  • 707
  • 1
  • 8
  • 15