0

In our servers there is one particular dependency (java jar), which makes a total of 1024 connection to various remote servers (the library works as a service discovery).

Intermittently, it is observed that all the connection in the connection pool gets exhausted and there are no connections available for the incoming traffic to that particular service. In such cases, restart of such service is the only way to free up the connections.

When such issues occur, netstat shows that a number of connections (1000+) are in ESTABLISHED state and have very high number in recvQ .

tcp    71464      0 localhost:57076 remote-host1:31200 ESTABLISHED
tcp    70512      0 localhost:47611 remote-host2:31200 ESTABLISHED
tcp    66184      0 localhost:44825 remote-host3:31200 ESTABLISHED
tcp    70512      0 localhost:40802 remote-host4:31200 ESTABLISHED
tcp    70016      0 localhost:58045 remote-host5:31200 ESTABLISHED
tcp    71464      0 localhost:35375 remote-host6:31200 ESTABLISHED
tcp    71464      0 localhost:48860 remote-host7:31200 ESTABLISHED

What could be the possible issue, that the connection are not receiving the data from remote host.

Pratyush
  • 101
  • 3
  • What version of Linux/Windows are you on? You seem to be facing port exhaustion and the number of ephemeral ports you have will be based on the version of your server. – James Shewey Mar 03 '15 at 14:47
  • We are using linux debian "Debian 3.2.60-1+deb7u3 x86_64" – Pratyush Mar 03 '15 at 14:48
  • The connection pool is of 1024. So the number of connection from the outbound server does not go beyond 1024. – Pratyush Mar 03 '15 at 14:49
  • The connection pool is of 1024, but suppose one connection fails or is idle for longer than the TCP timeout value. Does the daemon reconnoect? Also what do you get for "cat /proc/sys/net/ipv4/ip_local_port_range" on both the client and the server? – James Shewey Mar 03 '15 at 15:32

0 Answers0