how to make a persistent connection to redis or memcache within a twisted server

Question

I am trying to learn & understand how twisted works, for this I created a basic echo server, the one treats the input as the key for data stored on a redis server, if the input matches a key on the DB, it will print out the value, otherwise print 'key not found'.

I am avoiding the @defer.inlineCallbacks just to practice more how to work with Deferreds, the code is:

from twisted.internet import reactor, protocol
from txredis.client import RedisClient

REDIS_HOST = 'localhost'
REDIS_PORT = 6379


class Echo(protocol.Protocol):

    def connectionMade(self):
        self.factory.count += 1
        self.transport.write('There are %d clients\n' % self.factory.count)

    def dataReceived(self, data):
        clientCreator = protocol.ClientCreator(reactor, RedisClient)
        d = clientCreator.connectTCP(REDIS_HOST, REDIS_PORT)

        def cb_redis_get(r, data):
            d = r.get(data.strip())

            def cb_result(x):
                if x:
                    self.transport.write('%s\n' % x)
                else:
                    self.transport.write('key not found\n')

            d.addBoth(cb_result)
        d.addCallback(cb_redis_get, data)


class EchoFactory(protocol.ServerFactory):
    count = 0
    protocol = Echo


def main():
    reactor.listenTCP(1234, EchoFactory())
    reactor.run()

if __name__ == "__main__":
    main()

When a client connects telnet 0 1234 and enters a word, a connection to the redis server is made, with this approach, if I have 100 concurrent clients, the code will create 100 connections to the redis or memcache server.

This could be the expected behaviour but I would like to know, if could be posible to take advantage of the twisted reactor and when starting the server, create a persistent connection and use it for all the new connections, so that I could just have a single connection per instance and re-use it

If possible, any idea of how to properly implement it and also how to deal with reconnections?

Answer 1

Take a look at the txredisapi library. Its a quite complete twisted implementation of both individual connection and connection pools (persistent and non-persistent) for redis.

I've been using for a twisted project for a few months now and I've been quite happy with it.

---

Wait, ok maybe I missing what your trying to ask.

When I look at your code I see some odd things, like it looks like your spawning up a connection to redis on data receive from your telnet, that won't just create a connection per telnet, that will create a connection per data-flush on that telnet connection (This is almost certainly not what you want it to do).

Take a look at the examples with txredisapi they are more detailed then the examples I see with txredis

Roughly speaking what needs to happen is your logic should be split in two, one part should get the redis connection(s) live and the other part (which is basically what you have now) should process the telnet connections and push commands over the redis connection(s). The txredisapi lib will do all the redis work for you (... and txRedis may too, I don't know that lib well enough to know, the the txredisapi examples should help you either way).

(This style of question has come a a lot in the last few days, you might find a previous answer helpful to understand a common separation between parts in twisted, see: Persistent connection in twisted)