0

I have implemented a worker using Kombu's SimpleQueue. The implementation is as given below. When I run this worker for a few hours on a Ubuntu 16.04 system with redis as the backend, I notice a gradual memory build up on the process. When I run this worker for over a day, it ends up consuming all memory on the system and the system ends up being unusable, until the worker is killed.

On redis server, I have it configured with a timeout set to 5 seconds and a tcp_keepalive set to 60 seconds.

Worker Code:

from kombu import Connection

myqueue_name = 'test_queue'
backendURL = 'redis://127.0.0.1:6379/'

def GetConnection():
    conn = Connection(backendURL)

    return conn

def dequeue():
    conn = GetConnection()
    with conn:
        myqueue = conn.SimpleQueue(myqueue_name)

        item = None

        try:
            qItem = myqueue.get(block=True, timeout=2)
            item = qItem.payload
            qItem.ack()
        except Exception as e:
            qItem = None

        myqueue.close()

    conn.close()
    conn.release()

    return item

if __name__ == '__main__':
    try:
        i = 1
        while True:
            print 'Iteration %s: %s' % (i, dequeue())
            i = i + 1
    except (KeyboardInterrupt, SystemExit):
        print 'Terminating'

Here's a plot of free memory on the system:

enter image description here

What is going wrong here? Did I miss anything in the implementation?

Any help here will be greatly appreciated.

sradhakrishna
  • 99
  • 1
  • 5
  • What versions are you using? (redis, python) Although closed [this](https://github.com/celery/celery/issues/3436) seems related – Adonis Apr 06 '18 at 19:44
  • Python 2.7.12, Kombu 4.1.0, redis 3.0.6 – sradhakrishna Apr 06 '18 at 20:18
  • @Adonis the link seems to refer to a fix in celery - not in kombu/ redis. Trying to look for some fix in kombu - If you know of any, please let me know! – sradhakrishna Apr 06 '18 at 20:27
  • A couple of notes. If `no_ack` is not passed as keyword argument when instantiating the queue, it defaults to False and is acknowledged by the consumer. Also the context manager manages closing the connection on `__exit__`. – Oluwafemi Sule May 03 '18 at 19:27

0 Answers0