I have implemented a worker using Kombu's SimpleQueue. The implementation is as given below. When I run this worker for a few hours on a Ubuntu 16.04 system with redis as the backend, I notice a gradual memory build up on the process. When I run this worker for over a day, it ends up consuming all memory on the system and the system ends up being unusable, until the worker is killed.
On redis server, I have it configured with a timeout set to 5 seconds and a tcp_keepalive set to 60 seconds.
Worker Code:
from kombu import Connection
myqueue_name = 'test_queue'
backendURL = 'redis://127.0.0.1:6379/'
def GetConnection():
conn = Connection(backendURL)
return conn
def dequeue():
conn = GetConnection()
with conn:
myqueue = conn.SimpleQueue(myqueue_name)
item = None
try:
qItem = myqueue.get(block=True, timeout=2)
item = qItem.payload
qItem.ack()
except Exception as e:
qItem = None
myqueue.close()
conn.close()
conn.release()
return item
if __name__ == '__main__':
try:
i = 1
while True:
print 'Iteration %s: %s' % (i, dequeue())
i = i + 1
except (KeyboardInterrupt, SystemExit):
print 'Terminating'
Here's a plot of free memory on the system:
What is going wrong here? Did I miss anything in the implementation?
Any help here will be greatly appreciated.