6

We are running Celery behind Supervisor and start it with

celeryd --events --loglevel=INFO --concurrency=2

This, however, creates a process graph that is up to three layers deep and contains up to 7 celeryd processes (Supervisor spawns one celeryd, which spawns several others, which again spawn processes). Our machine has two CPU cores.

Are all of these processes working on tasks? Are maybe some of them just worker pools? How is the --concurrency setting connected to the number of processes actually spawned?

Hannes Struß
  • 1,560
  • 2
  • 13
  • 20

1 Answers1

4

You shouldn't have 7 processes if --concurrency is 2.

The actual processes started is:

  • The main consumer process

    Delegates work to the worker pool

  • The worker pool (this is the number that --concurrency decides)

So that is 3 processes with a concurrency of two.

In addition a very lightweight process used to clean up semaphores is started if force_execv is enabled (which it is by default i you're using some other transport than redis or rabbitmq).

NOTE that in some cases process listings also include threads. the worker may start several threads if using transports other than rabbitmq/redis, including one Mediator thread that is always started unless CELERY_DISABLE_RATE_LIMITS is enabled.

asksol
  • 19,129
  • 5
  • 61
  • 68
  • This doesn't match my experience. I have `--concurrency=1 --beat`, yet 4 celeryd processes are spawned. – Cerin Jun 12 '13 at 15:38
  • I use `DatabaseTransport` and Celery 3.0, and with `--concurrencry=1` I get 6 processes, similar as in OP case. – Gill Bates Mar 20 '14 at 12:08
  • @asksol I experience also 4 celeryd procs when --concurrency=1 and with two workers. I'm curious at to why that is, many thanks. – fpghost Apr 27 '17 at 20:26