3

We're running an spring boot (2.3.0) application on a tomcat server (9.0.24) and every so often we get an Too many open files exception, and the application stops working.

I checked the limits of the tomcat instance:

Max open files            4096                 4096                 files

Running lsof for the process returns a long list of sock files all looking like this:

java    17063 tomcat 4085u     sock                0,9       0t0  113306 protocol: TCP

root@popdb-dev:~# lsof -p 17063 | grep sock | wc -l
4003

However when I run ss to see what the sockets do I get a whole lot of closed sockets:

root@popdb-dev:~# ss -s
Total: 4384 (kernel 4485)
TCP:   4124 (estab 87, closed 4015, orphaned 0, synrecv 0, timewait 0/0), ports 0

Transport Total     IP        IPv6
*         4485      -         -
RAW       1         0         1
UDP       10        6         4
TCP       109       54        55
INET      120       60        60
FRAG      0         0         0

As you can see most of the sockets are already closed. I searched stackoverflow and google but couldn't find any reason why the closed sockets are not cleaned up. When I stop or restart tomcat the closed sessions are cleaned up.

What can we do so the sockets are cleaned up and our application doesn't stop working?

ebakker
  • 61
  • 3
  • Limits can be increased. But check https://stackoverflow.com/questions/15912370/how-do-i-remove-a-close-wait-socket-connection. OS parameters can be tuned to flush these connections faster. – Nic3500 Jul 18 '20 at 01:24
  • That seems to be about sockets with status CLOSE_WAIT. If I run ss -4 state CLOSE_WAIT and ss -6 state CLOSE_WAIT I only get 1 connection. so the rest of the closed connections don't show up. Not even when I run ss -4 state all. – ebakker Jul 20 '20 at 07:27
  • Any one found idea to clear the sockets? – sri Aug 02 '21 at 13:33

0 Answers0