How do I properly use connection pools in redis?

Question

It's not clear to me how connections pools work, and how to properly use them. I was hoping someone could elaborate. I've sketched out my use case below:

settings.py:

import redis

def get_redis_connection():
    return redis.StrictRedis(host='localhost', port=6379, db=0)

task1.py

import settings

connection = settings.get_redis_connection()

def do_something1():
    return connection.hgetall(...)

task2.py

import settings

connection = settings.get_redis_connection()

def do_something1():
    return connection.hgetall(...)

etc.

---

Basically I have a setting.py file that returns redis connections, and several different task files that get the redis connections, and then run operations. So each task file has its own redis instance (which presumably is very expensive). What's the best way of optimizing this process. Is it possible to use connection pools for this example? Is there a more efficient way of setting up this pattern?

For our system, we have over a dozen task files following this same pattern, and I've noticed our requests slowing down.

Thanks

Answer 1

Redis-py provides a connection pool for you from which you can retrieve a connection. Connection pools create a set of connections which you can use as needed (and when done - the connection is returned to the connection pool for further reuse). Trying to create connections on the fly without discarding them (i.e. not using a pool or not using the pool correctly) will leave you with way too many connections to redis (until you hit the connection limit).

You could choose to setup the connection pool in the init method and make the pool global (you can look at other options if uncomfortable with global).

redis_pool = None

def init():
    global redis_pool
    print("PID %d: initializing redis pool..." % os.getpid())
    redis_pool = redis.ConnectionPool(host='10.0.0.1', port=6379, db=0)

You can then retrieve the connection from a pool like this:

redis_conn = redis.Redis(connection_pool=redis_pool)

Also, I am assuming you are using hiredis along with redis-py as it should improve performance in certain cases. Have you also checked the number of connections open to the redis server with your existing setup as it most likely is quite high? You can use the INFO commmand to get that information:

redis-cli info

Check for the Clients section in which you will see the "connected_clients" field that will tell you how many connections you have open to the redis server at that instant.

Answer 2

You shall use a singleton( borg pattern ) based wrapper written over redis-py, which will provide a common connection pool to all your files. Whenever you use an object of this wrapper class, it will use the same connection pool.

REDIS_SERVER_CONF = {
    'servers' : {
      'main_server': {
        'HOST' : 'X.X.X.X',
        'PORT' : 6379 ,
        'DATABASE':0
    }
  }
}

import redis
class RedisWrapper(object):
    shared_state = {}

    def __init__(self):
        self.__dict__ = self.shared_state

    def redis_connect(self, server_key):
        redis_server_conf = settings.REDIS_SERVER_CONF['servers'][server_key]
        connection_pool = redis.ConnectionPool(host=redis_server_conf['HOST'], port=redis_server_conf['PORT'],
                                               db=redis_server_conf['DATABASE'])
        return redis.StrictRedis(connection_pool=connection_pool)

Usage:

r_server = RedisWrapper().redis_connect(server_key='main_server')
r_server.ping()

UPDATE

In case your files run as different processes, you will have to use a redis proxy which will pool the connections for you, and instead of connecting to redis directly, you will have to use the proxy. A very stable redis ( and memcached ) proxy is twemproxy created by twitter, with main purpose being reduction in open connections.

Answer 3

Here's a quote right from the Cheese Shop page.

Behind the scenes, redis-py uses a connection pool to manage connections to a Redis server. By default, each Redis instance you create will in turn create its own connection pool. You can override this behavior and use an existing connection pool by passing an already created connection pool instance to the connection_pool argument of the Redis class. You may choose to do this in order to implement client side sharding or have finer grain control of how connections are managed.

pool = redis.ConnectionPool(host='localhost', port=6379, db=0)
r = redis.Redis(connection_pool=pool)

Moreover, instances are thread-safe:

Redis client instances can safely be shared between threads. Internally, connection instances are only retrieved from the connection pool during command execution, and returned to the pool directly after. Command execution never modifies state on the client instance.

You say:

So each task file has its own redis instance (which presumably is very expensive). ... For our system, we have over a dozen task files following this same pattern, and I've noticed our requests slowing down.

It's quite unlikely that several dozens of connections can slow down Redis server. But because your code, behind the scenes, use connection pool, the problem is somewhere out of connections per se. Redis is in-memory storage, thus very fast in most imaginable cases. So I would rather look for the problem in the tasks.

Update

From comment of @user3813256. Yes, he uses connection pool at task level. The normal way to utilize built-in connection pool of redis package is just share the connection. In simplest way, your settings.py may look like this:

import redis

connection = None

def connect_to_redis():
    global connection
    connection = redis.StrictRedis(host='localhost', port=6379, db=0)

Then somewhere in bootstrapping of your application call connect_to_redis. Then use import connection in task modules.