3

I have one worker with a concurrency of 4. I see 4 processes started in flower and everything looks good.

If I do this in a shell then I see the 4 workers take the tasks and the rest are reserved and it processes 4 at a time until the queue is empty.

[my_task.apply_async() for i in xrange(10)]

However if I do this line by line only the first two tasks gets actively worked on and from then on out it processes only two at a time.

my_task.apply_async()
my_task.apply_async()
my_task.apply_async()
my_task.apply_async()
...

Any ideas?

Mingle
  • 846
  • 6
  • 13

1 Answers1

3

Usually this is because of subprocesses filling up the concurrency slots. Celery uses by default prefork as the execution pool, and everytime you spawn a subprocess of a task (another fork), it counts as a running process to fill up the concurrency slots.

The simplest way to avoid this is by using eventlet, which will allow you to spawn multiple async calls on each task. However, this requires that none of your tasks have blocking calls, such as subprocess.communicate, as they will block all tasks.

Otherwise, if you have necessary blocking calls, and you know your tasks only will have one running subprocess at a time, you can set the CELERYD_CONCURRENCY to the double (8) and set a start time-limit on your tasks, so 8 tasks won't get started right away (e.g. with @app.task(rate_limit='10/m')). However, this is bit of a hack, and using eventlet would definitely be preferred.

dwitvliet
  • 7,242
  • 7
  • 36
  • 62