3

The celery docs don't have too much information about the optimal number of celery workers to use per machine. I believe by default celery creates one worker per machine core.

I know from experimentation that on a single core machine, starting more celery workers is definitely beneficial (default is 1 worker because of 1 core). I'm looking for the threshold where adding more workers has marginally diminishing returns -- the optimal worker number per core. I am currently using a celery daemon with the daemon config file celeryd being having this line:

CELERYD_NODES="worker1 worker2 worker3"

My intention is to create 3 workers PER CORE (so if I started a 4 core machine, there would be 12 workers total). Am I doing this correctly, or will this only start 3 workers regardless of number of cores?

jeffrey
  • 3,196
  • 7
  • 26
  • 44
  • Celery won't worry about the number of workers you are starting for your tasks. Number of workers should depend on your tasks, their routing and the processing capacity of your machine. – Chillar Anand Oct 15 '14 at 09:11
  • I don't understand -- When I start 4 workers as opposed to 1, all of the workers are syncing and performing tasks. Wouldn't 4 here be faster than 1 then? – jeffrey Oct 15 '14 at 15:56
  • If you have computationaly intensive tasks, you cant; otherwise you might have. – Chillar Anand Oct 15 '14 at 20:19

0 Answers0