3

For multiprocessing pool, I could using the paramter '-c' to specify the number of worker processes. As the folloing shows.

celery worker -A celerytasks.celery_worker_init -c 5 --loglevel=info

But for gevent pool, the paramter '-c' is used to specify greenlet number. How can start multiple worker processes?

Or if I have to start mulitple wokers instead of one worker with multiple processes, could I do that with one command?

Jcyrss
  • 1,513
  • 3
  • 19
  • 31

1 Answers1

4

This command will run 1 worker with 5 pool processes. This means you have one worker able to execute 5 task at the time.

If you use gevent the only difference is that celery will use green threads instead of prefork to execute this tasks, but pay attention to the implication of this that you can read here (it's about eventlet but same concepts apply).

If you want to run multiple workers with different parameters (queue, concurrency) thant you need to use the celery multi command:

celery multi start 3 -c 3 -c:1 10
celery worker -n celery1@myhost -c 10
celery worker -n celery2@myhost -c 3
celery worker -n celery3@myhost -c 3
Mauro Rocco
  • 4,980
  • 1
  • 26
  • 40
  • 1
    So, if I use Superivisored to manage celery I have to specify the process number with "numprocs", instead of using some paramter in command line directly? – Jcyrss May 25 '16 at 00:13
  • 1
    Hi, here you find the answer, even if you may not like it. http://stackoverflow.com/questions/15558875/running-celeryd-multi-with-supervisor – Mauro Rocco May 25 '16 at 09:41
  • I end up use "numprocs" in supervisor configuration files, every thing looks fine till now. I don't want to use celery multi, because I think supervisor should not use command that already daemonlized a process. – Jcyrss May 25 '16 at 15:23
  • You may want to answer your own question or accept mine as correct than – Mauro Rocco May 25 '16 at 20:34