2

I am trying to set a different number of concurrent threads in Celery using --concurrency=1 (based on Workers Guide).

Anyway, when I run the daemon (celeryd ... --concurrency=1) I don't know how to see the number of threads that are running.

Do they correspond to each line shown in ps -ef?

fedorqui
  • 275,237
  • 103
  • 548
  • 598

2 Answers2

4

By default celery uses subprocesses for the workers. If you configure concurrency=1 you should have 2 processes:

  • the main process that consumes messages from the broker and manages the process poll
  • 1 worker process

In general you will have 1 main process + N worker subprocesses, as many as stated in --concurrency=N or CELERYD_CONCURRENCY=N

I'm not sure what happens with gevent workers.

enlavin
  • 161
  • 5
  • When you say `CELERYD_CONCURRENCY=N`, do you mean that this parameter can be put in celeryconfig.py? Thanks! – fedorqui May 23 '13 at 13:46
  • Yep, you can put it in celeryconfig.py or in the command line. If you use both the command line overrides the configuration file. – enlavin May 23 '13 at 14:47
  • What about the threads used for this process? How can we configure threads used for each process? In my case, where concurrency is 8 ps -ef | grep celery | grep -v grep | wc -l returns 8 and ps -eLf | grep celery | grep -v grep | wc -l returns 34 – Thomas John Jan 04 '19 at 09:34
1

ps -ef | grep celery | grep -v grep | wc -l returns the number of celery process running which is equal to the concurrency and ps -eLf | grep celery | grep -v grep | wc -l returns the number of threads used by celery

Thomas John
  • 2,138
  • 2
  • 22
  • 38