Celery does not release memory

Question

It looks like celery does not release memory after task finished. Every time a task finishes, there would be 5m-10m memory leak. So with thousands of tasks, soon it will use up all memory.

BROKER_URL = 'amqp://user@localhost:5672/vhost'
# CELERY_RESULT_BACKEND = 'amqp://user@localhost:5672/vhost'

CELERY_IMPORTS = (
    'tasks.tasks',
)

CELERY_IGNORE_RESULT = True
CELERY_DISABLE_RATE_LIMITS = True
# CELERY_ACKS_LATE = True
CELERY_TASK_RESULT_EXPIRES = 3600
# maximum time for a task to execute
CELERYD_TASK_TIME_LIMIT = 600
CELERY_DEFAULT_ROUTING_KEY = "default"
CELERY_DEFAULT_QUEUE = 'default'
CELERY_DEFAULT_EXCHANGE = "default"
CELERY_DEFAULT_EXCHANGE_TYPE = "direct"
# CELERYD_MAX_TASKS_PER_CHILD = 50
CELERY_DISABLE_RATE_LIMITS = True
CELERYD_CONCURRENCY = 2

Might be same with issue, but it does not has an answer: RabbitMQ/Celery/Django Memory Leak?

I am not using django, and my packages are:

Chameleon==2.11
Fabric==1.6.0
Mako==0.8.0
MarkupSafe==0.15
MySQL-python==1.2.4
Paste==1.7.5.1
PasteDeploy==1.5.0
SQLAlchemy==0.8.1
WebOb==1.2.3
altgraph==0.10.2
amqp==1.0.11
anyjson==0.3.3
argparse==1.2.1
billiard==2.7.3.28
biplist==0.5
celery==3.0.19
chaussette==0.9
distribute==0.6.34
flower==0.5.1
gevent==0.13.8
greenlet==0.4.1
kombu==2.5.10
macholib==1.5.1
objgraph==1.7.2
paramiko==1.10.1
pycrypto==2.6
pyes==0.20.0
pyramid==1.4.1
python-dateutil==2.1
redis==2.7.6
repoze.lru==0.6
requests==1.2.3
six==1.3.0
tornado==3.1
translationstring==1.1
urllib3==1.6
venusian==1.0a8
wsgiref==0.1.2
zope.deprecation==4.0.2
zope.interface==4.0.5

I just added a test task like, test_string is a big string, and it still has memory leak:

@celery.task(ignore_result=True)
def process_crash_xml(test_string, client_ip, request_timestamp):
    logger.info("%s %s" % (client_ip, request_timestamp))
    test = [test_string] * 5

are you using `virtualenv`? do you have the list of packages/versions you are using? — dnozay, Jul 09 '13 at 07:00

Erik Kalkoken · Answer 1 · 2023-03-11T13:09:06.583

There are two settings which can help you mitigate growing memory consumption of celery workers:

Max tasks per child setting (v2.0+):

With this option you can configure the maximum number of tasks a worker can execute before it’s replaced by a new process. This is useful if you have memory leaks you have no control over for example from closed source C extensions.

Max memory per child setting (v4.0+):

With this option you can configure the maximum amount of resident memory a worker can execute before it’s replaced by a new process. This is useful if you have memory leaks you have no control over for example from closed source C extensions.

However, those options only work with the default pool (prefork).

For safe guarding against memory leaks for threads and gevent pools you can add an utility process called memmon, which is part of the superlance extension to supervisor.

Memmon can monitor all running worker processes and will restart them automatically when they exceed a predefined memory limit.

Here is an example configuration for your supervisor.conf:

[eventlistener:memmon]
command=/path/to/memmon -p worker=512MB
events=TICK_60

Dechao Qiu · Accepted Answer · 2013-07-10T03:02:45.067

9

It was this config option that made my worker does not release memory.

CELERYD_TASK_TIME_LIMIT = 600

refer to: https://github.com/celery/celery/issues/1427

edited Jul 10 '13 at 03:02

answered Jul 10 '13 at 02:54

Dechao Qiu

429
1
3
9

score 4 · Answer 3 · edited Dec 29 '22 at 13:33

When you start your worker just set the max-tasks-per-child option like this to restart worker processes after every task:

celery -A app worker --loglevel=info --max-tasks-per-child=1

Here's the documentation:

https://docs.celeryproject.org/en/latest/userguide/workers.html#max-tasks-per-child-setting

With this option you can configure the maximum amount of resident memory a worker can execute before it’s replaced by a new process.

This is useful if you have memory leaks you have no control over for example from closed source C extensions.

The option can be set using the workers --max-memory-per-child argument or using the worker_max_memory_per_child setting.

score 1 · Answer 4 · answered Mar 14 '16 at 08:12

1

This was an issue in celery which I think is fixed.

Please refer: https://github.com/celery/celery/issues/2927

answered Mar 14 '16 at 08:12

A J

3,684
2
19
24

score 1 · Answer 5 · answered Jul 05 '17 at 18:35

1

set worker_max_tasks_per_child in your settings

answered Jul 05 '17 at 18:35

gizzmole

1,437
18
26

score 0 · Answer 6 · answered Jul 09 '13 at 06:59

0

You might be hitting this issue in librabbitmq. Please check whether or not Celery is using librabbitmq>=1.0.1.

A simple fix to try is: pip install librabbitmq>=1.0.1.

answered Jul 09 '13 at 06:59

dnozay

23,846
6
82
104

I am not using librabbitmq. Celery requires "amqp>=1.0.11,<1.1.0" to run. – Dechao Qiu Jul 09 '13 at 23:08

Celery does not release memory

6 Answers6

Linked