4

I know it is quite normal setting tcp_max_tw_buckets to a relatively small number such as 30000 or 50000, to avoid the situation when a host have a lots of time-wait state connections and application failed to open new one. It is something mentioned quite a lots. such as the question like this: How to reduce number of sockets in TIME_WAIT?

As before I know time-wait is a state to avoid TCP packets out of order, and it may be better using some other approach to coping it. And if you setting it to a small number thing may went wrong.

I feel I'm stucking at somewhere that I have to set tcp_max_tw_buckets to a small number, and don't know the specific scenarios I shall avoid it.

So my question is what is the side effect of setting tcp_max_tw_buckets to a very small value, and can I setup a specific scenario using lab environment, that a small number of tcp_max_tw_buckets will cause the trouble?

pingz
  • 555
  • 1
  • 10
  • 20

2 Answers2

4

As you can see in this Kernel source, that option prevents graceful termination of the socket. In terms of the socket state, you have reduced the time wait duration for this connection to zero.

So what happens next? First off, you'll see the error message on your server. The rest is then a race condition for subsequent connections from your clients. Section 2 of rfc 1337 then covers what you may see. In short, some connections may show the following symptoms.

  1. Corruption of your data stream (because the socket accepts an old transmission).
  2. Infinite ACK loops (due to an old duplicate ACK being picked up).
  3. Dropped connections (due to old data turning up in the SYN-SENT state).

However, proving this may be hard. As noted in the same RFC:

The three hazards H1, H2, and H3 have been demonstrated on a stock Sun OS 4.1.1 TCP running in an simulated environment that massively duplicates segments. This environment is far more hazardous than most real TCP's must cope with, and the conditions were carefully tuned to create the necessary conditions for the failures.

Community
  • 1
  • 1
Peter Brittain
  • 13,489
  • 3
  • 41
  • 57
2

The real answer to your question is that the correct way to avoid TIME_WAIT states is to be the end that receives the first close.

In the case of a server, that means that after you've sent the response you should loop waiting for another request on the same socket, with a read timeout of course, so that it is normally the client end which will close first. That way the TIME_WAIT state occurs at the client, where it is fairly harmless, as clients don't have lots of outbound connections.

user207421
  • 305,947
  • 44
  • 307
  • 483
  • My situation is a system have a lots of internal HTTP API, and the server, a PHP server, it is in fact both the server and the client, massive time-wait connections were generated as PHP requesting other internal API via HTTP, but being not closing it by a "Connection: close" HTTP header. So limiting tcp_max_tw_buckets is the best solution I can come with for now. – pingz Sep 06 '17 at 03:58
  • I didn't understand the part about 'connection: close', but your best solution is to allow both HTTP keepalive and connection pooling in the client PHP code, if it is supported. That will cause many fewer connections to exist, and they will tend to remain open rather than closed and going through TIME_WAIT. – user207421 Sep 06 '17 at 05:55
  • `Connection: close` is refered to [rfc2616 section 14.10 Connection](https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html), using that header client can release local opened ports, reduce the number of connections of time-wait state. I understand what you suggested is the ultimate solution, but I have to live with this situation quite long time due to limited resource and limited time, so I really need is a direct answer to my question. – pingz Sep 06 '17 at 06:27
  • I know what 'Connection: close' is. What I didn't understand is 'but being not closing it by a "Connection: close" HTTP header', which isn't even English. I can't even tell from that whether you are or aren't setting 'Connection: close'. – user207421 Sep 06 '17 at 09:43
  • Oh, sorry for that, what I mean is client didn't set `Connection: close`, and that is not going to be changed for a while. – pingz Sep 07 '17 at 08:36