3

How do you ensure celeryd only runs as a single process? When I run manage.py celeryd --concurrency=1 and then ps aux | grep celery I see 3 instances running:

www-data  8609  0.0  0.0  20744  1572 ?        S    13:42   0:00 python manage.py celeryd --concurrency=1
www-data  8625  0.0  1.7 325916 71372 ?        S    13:42   0:01 python manage.py celeryd --concurrency=1
www-data  8768  0.0  1.5 401460 64024 ?        S    13:42   0:00 python manage.py celeryd --concurrency=1

I've noticed a similar problem with celerybeat, which always runs as 2 processes.

Jamie Forrest
  • 10,895
  • 6
  • 51
  • 68
Cerin
  • 60,957
  • 96
  • 316
  • 522
  • How/where are you calling `python manager.py celeryd...`? Not sure if this might be the problem, but if you're django is running behind apache, and apache is running in multiprocess mode, each apache process may be starting it's own celeryd. – mklauber Jun 12 '13 at 18:59
  • @mklauber, I'm calling it via supervisord with command `python manage.py celeryd --concurrency=1` – Cerin Jun 12 '13 at 19:40
  • It looks like having n+1 processes is expected, as Celery launches a master and several worker subprocesses. So what you're seeing with celerybeat is expected, but then with --concurrency=1 you should still see 2 worker processes and not 3. Which version of Celery are you using? No issue on my end with Celery 3.0.19. Also, does Celery tell you "concurrency: 1 (processes)" somewhere on stdout? – Nicolas Cortot Jun 17 '13 at 22:19
  • I am having a similar issue. I even tried setting --maxtasksperchild=1, but still get 3 python processes when I start the celeryd daemon. – Jamie Forrest Jun 19 '13 at 15:36

1 Answers1

0

As per this link .. The number of processes would be 4: one main process, two child processes and one celerybeat process, also if you're using FORCE_EXECV there's another process started to cleanup semaphores.

If you use celery+django-celery development, and using RabbitMQ or Redis as a broker, then it shouldn't use more than one extra thread (none if CELERY_DISABLE_RATE_LIMITS is set)

securecurve
  • 5,589
  • 5
  • 45
  • 80