0

A bizarre problem on a production setup.

Scenario

1) A server listens on a particular port.

2) There are potentially thousands of TCP connection requests.

3) About 200-300 TCP connections reach ESTABLISHED state (netstat output).

4) After this, the connections begin to drop off (and sometimes stay in CLOSE_WAIT state indefinitely and clear off only at system reboot).

5) The new connection requests (SYN) gets no response back (SYN ACK).

6) After the initial build up and drop off, the system shows zero ESTABLISHED state connections and stays there up until a reboot after which the cycle repeats.

Does not look like a DDoS attack (ip addresses all good. TCP port correct as well). Has someone experienced such a scenario?

Note:

1) Even after several reboots, the issue persists. The setup worked perfectly fine before.

2) The firewall setting looks good.

3) The devices reach the server through a security gateway.

4) The workaround for this issue is to control the number of TCP connection requests per minute (throttling set on the OS - redhat)

HBruijn
  • 77,029
  • 24
  • 135
  • 201
TheMonkWhoSoldHisCode
  • 135
  • 5
  • Try this link.Might be of some help to you http://blog.stephencleary.com/2009/05/detection-of-half-open-dropped.html –  Mar 14 '15 at 16:58
  • Can we get more details on the server platform, application, and use case here? It's unclear to me what the app is doing. – Mike B Mar 14 '15 at 19:23
  • @MikeB The server is a RedHat5.5 which hosts a JEE server (glassfish) and the TCP listener application is deployed in the glassfish. This application is a Network Management Server. I call this as local endpoint. The remote endpoints are hardware devices which connect to the NMS for configuration management. – TheMonkWhoSoldHisCode Mar 16 '15 at 06:47
  • Anything in dmesg ? Are the remote peers behind a NAT ? – Xavier Lucas Mar 16 '15 at 11:53
  • @XavierLucas I don't have dmesg logs from the production setup. Is there anything specific to look in to dmesg logs? The remote peers are behind a security gateway. – TheMonkWhoSoldHisCode Mar 17 '15 at 05:28

0 Answers0