Too many TIME_WAIT connections with Jetty

Question

I am running an API on 10 different servers, all of them are behind a firewall. I am using jetty 8 to serve all the http requests. The use case for this API is short lived connections.

A few month ago I started to get random Too many open file descriptors errors. These errors make the server completely unresponsive and I need to restart the jetty server in order to fix that. Today this happened 0-10 times a day depending on the traffic I am getting.

After some investigations, I noticed that I am exhausting the number of available connections because all of them are stuck in the TIME_WAIT state so I can't create new ones.

ss -s

TCP:   13392 (estab 1549, closed 11439, orphaned 9, synrecv 0, timewait *11438*/0), ports 932

On this example the number of connections in TIME_WAIT state is pretty low but it can go up to 50k.

I have been trying several kernel tweaks and I also tried to set the SO_LINGER timer to 1 second for jetty sockets. All these changes helped reduce the frequency but I am still getting errors regularly.

Also worth mentioning, I am receiving around 3k requests/second on each server and the cpu usage is very low. The bottleneck to scale my traffic today is this connection issue.

Does anyone have an idea of what I can do to handle that correctly ?

Did you take a thread dump of your threads to see where are they waiting during these peak periods? — dcernahoschi, Jan 03 '14 at 09:14
@dcernahoschi These are process states and port states, not thread states. Your question is irrelevant. — user207421, Jan 03 '14 at 10:20

user207421 · Answer 1 · 2014-12-17T07:18:13.790

1

'Too many open file descriptors' is probably caused by a resource leak in your application.

The TIME_WAIT state is caused by being the end that first sends a close, instead of the end that first receives the close. You might want to reconsider your application protocol so that it is the client which closes first. This is not too hard to arrange. It falls out free if you use client-side connection pooling for example.

These two conditions are not related. The TIME_WAIT state can only occur on a port whose socket has already been closed. It does not cause 'too many open file descriptors' problems.

edited Dec 17 '14 at 07:18

answered Jan 03 '14 at 10:18

user207421

305,947
44
307
483

Doesn't TIME_WAIT mean that I am the one initiating the close? – MichelT Jan 03 '14 at 16:59
I could have that back to front but it doesn't affect the irrelevance of this to your FD problem. – user207421 Jan 03 '14 at 19:09
Who close first reach the TIME-WAIT state. – Jay Dec 17 '14 at 06:47
@Jay Thanks, you are correct, I should have fixed this ages ago, done now. – user207421 Dec 17 '14 at 07:31

Too many TIME_WAIT connections with Jetty

1 Answers1