Django Celery Memory not release

Question

In my django project I have the following dependencies:

django==1.5.4
django-celery==3.1.9
amqp==1.4.3
kombu==3.0.14
librabbitmq==1.0.3 (as suggested by https://stackoverflow.com/a/17541942/1452356)

In dev_settings.py:

DEBUG = False
BROKER_URL = "django://"
import djcelery
djcelery.setup_loader()
CELERYBEAT_SCHEDULER = "djcelery.schedulers.DatabaseScheduler"
CELERYD_CONCURRENCY = 2
# CELERYD_TASK_TIME_LIMIT = 10

CELERYD_TASK_TIME_LIMIT is commented as suggested here https://stackoverflow.com/a/17561747/1452356 along with debug_toolbar as suggested by https://stackoverflow.com/a/19931261/1452356

I start my worker in a shell with:

./manage.py celeryd --settings=dev_settings

Then I send a task:

class ExempleTask(Task):

  def run(self, piProjectId):
    table = []
    for i in range(50000000):
      table.append(1)
    return None

Using a django command:

class Command(BaseCommand):

  def handle(self, *plArgs, **pdKwargs):
    loResult = ExempleTask.delay(1)
    loResult.get()
    return None

With:

./manage.py purge_and_delete_test --settings=dev_settings

I monitor the memory usage with:

watch -n 1 'ps ax  -o rss,user,command | sort -nr | grep celery |head -n 5'

Every time I call the task, it increase the memory consumption of the celeryd/worker process, proportionally to the amount of data allocated in it...

It seems like a common issue (c.f. others stackoverflow link), however I couldn't fix it, even with the latest dependencies.

Thanks.

score 5 · Answer 1 · edited May 23 '17 at 12:30

This is a Python and OS issue, not really a django or celery issue. Without getting too deep:

1) A process will never free memory addressing space once it has requested it from the OS. It never says "hey, I'm done here, you can have it back". In the example you've given, I'd expect the process size to grow for a while, and then stabilize, possibly at a high base line. After your example allocation, you might call the gc interface to force a garbage collect to see how

2) This isn't usually a problem, because unused pages are paged out by the OS because your process stops accessing that address space that it has deallocated.

3) It is a problem if your process is leaking object references, preventing python from garbage collecting to re-appropriate the space for later reuse by that process, and requiring your process to ask for more address space from the OS. At some point, the OS cries uncle and will (probably) kill your process with its oomkiller or similar mechanism.

4) If you are leaking, either fix the leak or set CELERYD_MAX_TASKS_PER_CHILD, and your child processes will (probably) commit suicide before upsetting the OS.

This is a good general discussion on Python's memory management: CPython memory allocation

And a few minor things: Use xrange not range - range will generate all values then iterate over that list. xrange is just a generator. Have set Django DEBUG=False?

1) I already try the gc.collect() but It doesn't change anything 2) Here it's a problem because the system never deallocate this "unused" paged memory (Ubuntu 12.04.3 LTS) 3) The workers are never kill 4) Indeed it could be a solution however I want my process to finish his task... Yes I'm using DEBUG=False — Adrien, Apr 09 '14 at 08:42
@adrien regarding 4) max tasks per child will lead to a child's exit only after the tasks are completed. It will never abort mid-processing. — Nino Walker, Apr 20 '14 at 22:54

Django Celery Memory not release

1 Answers1

Linked