Python secure websocket memory consumption

Question

I am writing a web socket server in python. I have tried the approach below with txws, autobahn, and tornado, all with similar results.

I seem to have massive memory consumption with secure websockets and I cannot figure out where or why this might be happening. Below is an example in tornado, but I can provide examples in autobahn or txws.

import tornado.httpserver
import tornado.websocket
import tornado.ioloop
import tornado.web
import json 

class AuthHandler(tornado.websocket.WebSocketHandler):

    def open(self):
        print 'new connection for auth'

    def on_message(self, message):
        message = json.loads(message)
        client_id = message['client_id']
        if client_id not in app.clients:
            app.clients[client_id] = self
        self.write_message('Agent Recorded')

    def on_close(self):
        print 'auth connection closed'


class MsgHandler(tornado.websocket.WebSocketHandler):

    def open(self):
        print 'new connection for msg'

    def on_message(self, message):
        message = json.loads(message)
        to_client = message['client_id']
        if to_client in app.clients:
            app.clients[to_client].write_message('You got a message')

    def on_close(self):
        print 'msg connection closed'


app = tornado.web.Application([
    (r'/auth', AuthHandler),
    (r'/msg', MsgHandler)
])

app.clients = {}


if __name__ == "__main__":
    http_server = tornado.httpserver.HTTPServer(app, ssl_options={
        'certfile': 'tests/keys/server.crt',
        'keyfile': 'tests/keys/server.key'
    })
    http_server.listen(8000)
    tornado.ioloop.IOLoop.instance().start()

After making around 10,000 connections I find I am using around 700MB of memory with SSL compared to 43MB without, and I never get it back unless I kill the process. It seems like the problem is closely tied to the amount of connections made rather than messages sent.

The consumption seems to happen independent of the client (I wrote my own client and tried other clients).

Are secure websockets really that much more memory intensive that plain websockets? Or is my server code not implementing it correctly?

OpenSSL allocates a lot of memory for each connection by default. Some great details in this blog post - https://journal.paul.querna.org/articles/2011/04/05/openssl-memory-use/ — Jean-Paul Calderone, Nov 19 '13 at 18:30
Awesome! That article answered so many questions and the contained solutions have really brought down memory consumption. — sambev, Nov 19 '13 at 19:58

score 2 · Answer 1 · answered Feb 04 '14 at 15:17

2

I think the best solution is to use a real webserver (nginx apache) as a proxy and let it manage the ssl layer.

answered Feb 04 '14 at 15:17

julienfr112

2,077
2
25
37

Why don't you believe that Python web servers are "real"? – Glyph Feb 10 '14 at 16:09
Touché !! I meant real in the sense that have modules that supports all what is neaded in a http server open to the world (logging, DoS prevention, load balancing, compiled, ) – julienfr112 Feb 17 '14 at 14:47
1

+1 for @JulienFr, Python should not be directly exposed to the public network. Apache / Nginx are written in lower level languages and heavily optimized for the single task they have. – Ron E Jun 19 '14 at 06:37

Python secure websocket memory consumption

1 Answers1