2

There's a certain condition where I'm unable to receive a response from an HTTP request to a particular service: the request is sent through a load balancer and the response is too big to fit into a single packet. Under those conditions, a response is never received. Neither is an error. The server sends a TCP ACK in response to the packet containing the HTTP request, but nothing comes after that. The connection just hangs waiting for something to happen. The behavior is observed whether the request is sent via CURL, node's HTTP library, or Postman.

What could cause that behavior? How can I debug what is causing the issue?

  • I have verified that the service behind the load balancer is receiving and responding to the HTTP request, but the response is getting lost somewhere.
  • We have other services with a similar setup that don't exhibit this issue.
  • We tried setting a different instance of the load balancer but it had the same problem.

In tshark on the server, I see a handful of [TCP Retransmission] entries after it receives one of the problematic requests through the load balancer (this doesn't happen when the same request is received directly). This seems to indicate that something between the client and the server is misdirecting TCP traffic (highly implicates the load balancer) but I can't understand what would cause that to occur only when HTTP responses are split into multiple packets, and only by this particular service.

Conditions that cause the failure:

Direct Loadbalancer
Single Packet ✅ ✅
Multiple Packets ✅

Network topology:

Client -> Load Balancer -> Service
Justin York
  • 151
  • 7
  • Does this load balancer have logs that could be used to understand what happens? – berndbausch Mar 09 '21 at 22:10
  • @berndbausch We haven't found any yet. It's part of a Fortigate firewall appliance. – Justin York Mar 09 '21 at 22:13
  • 1
    My point is, your client sends requests, they arrive at the HTTP server, which sends responses as expected, which responses are not relayed back to the client. Ergo the LB is at fault, and you should focus your troubleshooting activities there. How to troubleshoot this depends on the particular LB used. – berndbausch Mar 09 '21 at 23:25
  • A PMTU blackhole sounds a lot like what I'm experiencing: https://en.wikipedia.org/wiki/Black_hole_(networking) – Justin York Mar 10 '21 at 20:18
  • I have confirmed that ICMP packets type 3 seem to be dropped which lends to the idea of a PMTU black hole. It's probably not a problem with the load balancer. Instead, it appears to be a problem with network configuration. – Justin York Mar 10 '21 at 22:19
  • 1
    Thanks, I learned something. – berndbausch Mar 10 '21 at 23:35

1 Answers1

1

It turns out that I discovered a PMTUD black hole.

I first suspected a fragmentation problem due to an article from Cloudflare about packet fragmentation. I confirmed the issue using ping to send different packet sizes: ping -s SIZE.

First I tried 1500 which is the default MTU for ethernet.

> ping 172.16.5.1 -s 1500

But nothing happened. It just hung and printed nothing. If there's a problem, you're supposed to get an error like this:

ping: sendto: Message too long
Request timeout for icmp_seq 0

Eventually, I found an article that described the issue in more detail and recommended a fix: setting /proc/sys/net/ipv4/tcp_mtu_probing to 1 to enable black hole discovery (which is not enabled by default in Linux ‍♂️). I wasn't actually able to modify that file, so I followed the recommendation to set it in /etc/sysctl.conf and then reload settings via sysctl --system. After all that, I was able to receive responses from the server.

MTU probing adds overhead to the response (it was noticeably delayed) so I still intend to figure out why ICMP packets are not being received, but at least I now understand the problem and have a workable solution.

Update: Turns out a LAG had an MTU of 1500 so it was dropping the packets.

Justin York
  • 151
  • 7