1

I am using a websocket server coded in Python 3.5. This is needed to provide my website realtime information.

The "server"-library I use is called tornado (version 4.3) which handles the websocket connection and http requests. For backwards compatibility (if a browser does not support websocket yet) I use the python library sockjs-tornado (version 1.0.3) which also adds some additional features to the websocket connection.

This connection uses the protocol wss:// so it is encrypted and loads a certificate for the socket connections. The Problem is that I get an error when the server has been running for quite a while.

ERROR:tornado.application:Exception in callback (<socket.socket fd=18, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('***', 8443)>, <function wrap.<locals>.null_wrapper at 0x6fbeea4769d8>)
Traceback (most recent call last):
  File "/home/website/python/tornado/ioloop.py", line 883, in start
    handler_func(fd_obj, events)
  File "/home/website/python/tornado/stack_context.py", line 275, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/website/python/tornado/netutil.py", line 274, in accept_handler
    callback(connection, address)
  File "/home/website/python/tornado/tcpserver.py", line 239, in _handle_connection
    do_handshake_on_connect=False)
  File "/home/website/python/tornado/netutil.py", line 510, in ssl_wrap_socket
    context = ssl_options_to_context(ssl_options)
  File "/home/website/python/tornado/netutil.py", line 487, in ssl_options_to_context
    context.load_cert_chain(ssl_options['certfile'], ssl_options.get('keyfile', None))
OSError: [Errno 24] Too many open files

I already increased the limit of open files on the linux server, but if the python script was not restarted since 24h then these errors will pop up if about 300 clients are connected. If I restart the script during this time everything will work fine again, even after everyone reconnected.

I do not really know what to do because it seems like that tornado is not closing the certificate files correctly and loading them again on every new connection.

DoNotClick
  • 66
  • 1
  • 8

1 Answers1

2

It may not be the case that tornado is leaking file handles; it's just tornado that is hitting the limit after a while because it regularly opens files.

Get the PID for the process in question and look at /proc/[PID]/fd. It will list all the file handles that are currently open. If your code keeps files open due to some reason, they will pile up there.

user2722968
  • 13,636
  • 2
  • 46
  • 67
  • I tried this and I also made statistics about all processes and the file descriptors they use. I watched the python script for quite a while now and the files which are opened increase constantly. If I use the command lsof -p [pid] I see a lot of similar rows which look something like this: python3.5 27598 root 111u IPv4 18662513 0t0 TCP ns000.ip-000-000-000.eu:pcsync-https->000.000.000.000:25467 (ESTABLISHED). Are this (leaked) websocket connections of tornado? – DoNotClick Apr 27 '17 at 20:23
  • well those sockets are not exactly leaking as they are `ESTABLISHED`, that is, still open and ready for business. Maybe your websocket library does close the tcp connection after the websocket went out of business? – user2722968 Apr 27 '17 at 21:25
  • I have some kind of online counter on my website which counts the array of all current connections. If library calls the on_open and on_close event the specific connection is removed or added to the array, so I can see how many clients are currently online. Most of the time this number is between 100 - 300 and does not seem to contain any dead connections, but the tcp connections of the process increase all the time. Also the open and close events are called correctly, so it does not make sense that connections are not closed or is it maybe just a sockjs-tornado bug? – DoNotClick Apr 28 '17 at 12:40
  • It seems to be a known problem: https://groups.google.com/forum/#!topic/sockjs/DvnvKPybrD4 – user2722968 Apr 28 '17 at 14:46
  • So how can we hold more than 1024 websocket connections? After all, this is the purpose of a websocket, to be lightweight and persistent. I can't have more than 1024 connections with my python tornado – Nathan B Feb 07 '22 at 07:55