23

Is it possible to set the concurrency (the number of simultaneous workers) on a per-task level in Celery? I'm looking for something more fine-grained that CELERYD_CONCURRENCY (that sets the concurrency for the whole daemon).

The usage scenario is: I have a single celerlyd running different types of tasks with very different performance characteristics - some are fast, some very slow. For some I'd like to do as many as I can as quickly as I can, for others I'd like to ensure only one instance is running at any time (ie. concurrency of 1).

Parand
  • 102,950
  • 48
  • 151
  • 186

1 Answers1

32

You can use automatic routing to route tasks to different queues which will be processed by celery workers with different concurrency levels.

celeryd-multi start fast slow -c:slow 3 -c:fast 5

This command launches 2 celery workers listening fast and slow queues with 3 and 5 concurrency levels respectively.

CELERY_ROUTES = {"tasks.a": {"queue": "slow"}, "tasks.b": {"queue": "fast"}}

The tasks with type tasks.a will be processed by slow queue and tasks.b tasks by fast queue respectively.

uglide
  • 1,695
  • 14
  • 19
mher
  • 10,508
  • 2
  • 35
  • 27
  • 1
    Thanks 0x00mh. So I can define concurrency on queues, but not on tasks. And I believe this means I'm starting multiple celery daemons. So I guess that means there's no way to set the per-task concurrency without using a separate daemon? – Parand Feb 08 '12 at 20:08
  • CELERYD_CONCURRENCY defines how many processes (worker processes) to launch. Worker processes consume messages from the broker independently. Message contains the task name to execute. – mher Feb 09 '12 at 10:16