11

We are using

django-celery==3.1.10
celery==3.1.20
python 2.7.13

We have written a CustomDataBaseScheduler to schedule task, which schedules the task perfectly on time. We are running CeleryBeat Process as init script, but celeryBeat consumes Full memory of the system i.e. 24GB in a day.

I tried to run pmap on celerybeat Process, but it shows [anon] has took the most memory.

Can someone please help to debug and fix this.

Ivan Kolesnikov
  • 1,787
  • 1
  • 29
  • 45
Virender Dubey
  • 196
  • 13
  • Did you look into any options of timeout or so, like `CELERYD_TASK_TIME_LIMIT`? https://stackoverflow.com/questions/17541452/celery-does-not-release-memory – Nagaraj Tantri Aug 01 '17 at 11:34
  • Any chance you are running with debug = True on settings.py? It is known that this causes a memory leak on celery: https://groups.google.com/forum/#!topic/celery-users/smV5tw59Ia4 – Alex Aug 01 '17 at 13:34
  • Migrate to http://serverfault.com/ ? – Spacedman Aug 02 '17 at 08:15
  • does your debug turned on? – Erdenezul Aug 02 '17 at 13:40
  • This Parameter is for CELERY Workers.. Although have tried with .. ######## Celery Settings CELERY_TASK_RESULT_EXPIRES=120 # 2 minutes CELERY_RESULT_BACKEND="amqp" CELERY_IGNORE_RESULT=True CELERY_SEND_EVENTS=False CELERY_EVENT_QUEUE_EXPIRES=60 but invain – Virender Dubey Aug 04 '17 at 12:16

1 Answers1

3

First of all if you are on django 1.8 or above please use celery 4.0 and above. In that case you will not require django-celery. Also in that case follow this tutorial. http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html

Coming to your problem it may be one of the following reasons:

  1. Your workers are overloaded. Try using concurrency as mentioned here
  2. Check if your django settings DEBUG is set to True. It can cause memory leakages and celery advises against it when you run it.
  3. Check for memory leak fixes in the history. To me it seemed the next version after yours, 3.1.21 to be precise has some memory leak fixes. Try upgrading to the latest 3.x version(only if you cannot use 4.x for any reason.)
  4. If all else fails, try some monitoring tools to debug the scenario about what is happening. Some monitoring tools are mentioned here.
rmad17
  • 164
  • 7
  • django is 1.7, so cannot use celery 4. debug is True in settings.py but we use init scripts to start celery where we have used INFO not debug. – Virender Dubey Aug 03 '17 at 06:20
  • I meant `debug` in django settings.py. When you run celery with django `DEBUG=True` in settings.py, you will get [this](https://github.com/celery/celery/blob/d90caee6d91a0fcc91756329503a35bf8fef720a/celery/fixups/django.py#L202). Switch off debug and check if that works. – rmad17 Aug 03 '17 at 07:09
  • tried upgrading django-celery==3.2.1 but it requires upgrading of django>=1.8 and we have django==1.7 So upgraded django-celery==3.1.17 which is latest in 3.1. Also will try with debug=False. – Virender Dubey Aug 04 '17 at 12:18
  • That's great, thanks to @rmad17, turning off debug in django settings.py fixes the memory issue cause by celery beat.. – Virender Dubey Sep 15 '17 at 13:48