1

We've been trying to get a grip on a really weird problem where we can wget a page from Apache 2.2.19 on solaris 10 and some permutations of requests reliably take various fixed lengths of time to respond.

It looks to be based around the closing of the TCP socket, from a tcpdump on the client, we typically see a pause between the server filling the TCP window with its response for the final time, and the final lump of data along with the FIN from the server.

So on the wire it just hangs mid transfer of the HTTP response, but a netstat on the server shows a socket in FIN_WAIT_1. we can't do a tcpdump on the server to clarify but it looks to us like the the OS has handed off the TCP conversation to the hardware, so believes it's started the 4way close, yet the NIC never puts that packet, nor the outstanding data packets (maybe 1 or 2 @ 1500 bytes and a straggler of 400 or whatever).

That's about as clean a picture as we can paint, apart from one recent test we did which looked amazing... a file of 64076 bytes is served by apache - 432ms delay for the last packet. we add ONE char to the file, and that caused an additional packet to be created, 53 bytes which also included the FIN, as opposed to the empty 52 byte FIN we see on the slightly smaller file, the one with the delay. The existence of this extra packet was reliably changing how the FIN conversation happened, and made the conversation go from almost half a second to a few milliseconds.

Most typically over the life of out investigations, this delay has been 4.6 seconds, again we see the window size fluctuate, SACK's going back to Apache when needed, but the LAST time that window gets full, it hangs for 46. seconds, and then back comes a final 2 or 3kb of data, and the FIN that Solaris thinks it sent out ages ago.

Our tcpdumps are on a F5 BigIP, so there is an ASIC for the traffic to traverse, as well as a Cisco 6509 (L2 only), however we do see the same user experience when doing a wget on a neighboring Solaris box, so don't believe it's any black magic that the BigIP is doing.

But this is all mixed up with confusion about window sizes and MSS's and the likes, but if this is sounding familiar to anyone, we're all ears!

Chris Phillips
  • 254
  • 4
  • 15
  • What are the NIC(s) on the server? How are they configured? Are you running GB ethernet with autonegotiation disabled - that's explicitly outside the specs: *Autonegotiation is a requirement for using 1000BASE-T[12] according to Section 28D.5 Extensions required for Clause40 (1000BASE-T).[13] At least the clock source has to be negotiated, as one endpoint must be master and the other endpoint must be slave.* (https://en.wikipedia.org/wiki/Gigabit_Ethernet#1000BASE-T) – Andrew Henle Dec 12 '15 at 16:18
  • I don't know what the NICs are on the box, but they're configured with negotiation at 1gig. it's ONLY this scenario where things go awry, so wouldn't have thought negotiations could be possible - Ethernet in itself is fine. thanks – Chris Phillips Dec 16 '15 at 11:47
  • What version of Solaris? If Solaris 10, you can see the NICs with `dladm show-dev`, on Solaris 11 use `dladm show-phys`. I'm not sure what you mean by "configured with negotiation at 1gig". Autonegotiation must be set on both the server and the switch. There are way too many network admins who do not understand that autonegotiation is part of the 1000baseT spec (and higher). Since turning off autonegotiation is outside the spec, no vendor on the planet pays to test their equipment under those conditions, nor will they provide any support if you run with autonegotiation disabled. – Andrew Henle Dec 16 '15 at 21:41

0 Answers0