6

After upgrading Django to 1.6, my celery worker is eating up RAM. Seems that the memory allocated for the workers isn't released and grows after every task.

Related Settings:

# DB:
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'somedb',
        'USER': '',
        'PASSWORD': '',
        'HOST': 'localhost',
        'PORT': '',
    }
}


# CELERY SETTINGS:
CELERY_RESULT_BACKEND = 'redis://'
BROKER_URL = 'redis://'

Related package versions:

Django==1.6
celery==3.0.24
django-celery==3.0.23
billiard==2.7.3.34
kombu==2.5.16
redis==2.7.6

Happens in both my local env (with DEBUG=False) running the worker manually and in a staging environment where celery is running with Upstart.

Updates:

  1. Tried setting autocommit=False with no success.
  2. Could be it's not related to the Django version upgrade, but to some setting or 3rd party package that I had to upgrade to make the switch to 1.6.
yprez
  • 14,854
  • 11
  • 55
  • 70
  • Do you have time-limits set? http://docs.celeryproject.org/en/latest/userguide/workers.html#time-limits – nickzam Nov 12 '13 at 12:14
  • @nikzam, no, maybe I should. The tasks are short, the longest ones take 30 seconds. – yprez Nov 12 '13 at 12:15
  • 2
    You could try upgrading to django-celery 3.1 maybe, the test suite for djcelery 3.0.24 was not passing with Django 1.6 – asksol Nov 12 '13 at 13:15
  • @asksol, it doesn't seem to be a Celery bug (will post answer/update soon), and the 3.1 version gave me trouble with Django settings, so the upgrade wasn't very smooth. But thanks, I'll definitely upgrade to 3.1 at some point to avoid future problems. – yprez Nov 12 '13 at 13:56

2 Answers2

7

It turns out the memory leak was not directly caused by the Django upgrade or Celery.

After a lot of digging around I found that, surprisingly, the celery worker memory leak happens because I upgraded django-debug-toolbar from 0.9.4 to 0.11.0 (which is needed for Django 1.6 compatibility).

Still no idea what exactly caused this issue, or why it only happens in the celery worker processes and not in the app server ones (Gunicorn).

Removing django-debug-toolbar from the installed apps and middleware solves the issue. At least temporarily.

yprez
  • 14,854
  • 11
  • 55
  • 70
1

It appears that the change from django-debug-toolbar 0.9.4 to 0.11.0 did introduce a memory leak caused by the LoggingPanel storing an infinite number of messages. If you had a process that was using the logging subsystem, it's likely you ran into this problem. You can also remove the LoggingPanel from the list of default panels to work around the problem.

Apparently in 0.9.4, the panels were lazily initialized only when the middleware was accessed. This changed in 0.11.0: the panels are initialized immediately after import, and the LoggingPanel module was intercepting all logs regardless whether DEBUG was set.

I've submitted a fix for this bug.

stanhu
  • 136
  • 1
  • 5
  • 23