Celery: how to separate different environments with different workers?

Question

I need to route all tasks of a certain django site instance to a certain queue. My setup is as following:

several webservers running a Django project (1.7)
one server running celery workers (3.1.7)
Three environments: production, staging, development. Each environment runs with a different DJANGO_SETTINGS_MODULE, with a different CELERY_DEFAULT_QUEUE setting.
One redis instance as broker (everything in the same database)

On the "celery server", I run multiple worker instances through supervisor (simplified conf):

[program:production_queue]
environment=PYTHONPATH=/pth/to/src/:/pth/to/site-packages/,DJANGO_SETTINGS_MODULE=website.settings.production
command=/pth/to/python celery -A website.celery worker --events --queues myserver --loglevel WARNING --concurrency 4 -n production@celery.myserver.nl

[program:staging_queue]
environment=PYTHONPATH=/pth/to/src/:/pth/to/site-packages/,DJANGO_SETTINGS_MODULE=website.settings.staging
command=/pth/to/python celery -A website.celery worker --events --queues myserver_staging --loglevel WARNING --concurrency 1 -n staging@celery.myserver.nl

[program:development_queue]
environment=PYTHONPATH=/pth/to/src/:/pth/to/site-packages/,DJANGO_SETTINGS_MODULE=website.settings.development
command=/pth/to/python celery -A website.celery worker --events --queues myserver_development --loglevel INFO --concurrency 1 -n development@celery.myserver.nl

This works, with inspection:

$ celery -A website.celery inspect activeues
-> production@celery.myserver.nl: OK
    * {u'exclusive': False, u'name': u'myserver', u'exchange': {u'name': u'celery', u'durable': True, u'delivery_mode': 2, u'passive': False, u'arguments': None, u'type': u'direct', u'auto_delete': False}, u'durable': True, u'routing_key': u'celery', u'no_ack': False, u'alias': None, u'queue_arguments': None, u'binding_arguments': None, u'bindings': [], u'auto_delete': False}
-> staging@celery.myserver.nl: OK
    * {u'exclusive': False, u'name': u'myserver_staging', u'exchange': {u'name': u'celery', u'durable': True, u'delivery_mode': 2, u'passive': False, u'arguments': None, u'type': u'direct', u'auto_delete': False}, u'durable': True, u'routing_key': u'celery', u'no_ack': False, u'alias': None, u'queue_arguments': None, u'binding_arguments': None, u'bindings': [], u'auto_delete': False}
-> development@celery.myserver.nl: OK
    * {u'exclusive': False, u'name': u'myserver_development', u'exchange': {u'name': u'celery', u'durable': True, u'delivery_mode': 2, u'passive': False, u'arguments': None, u'type': u'direct', u'auto_delete': False}, u'durable': True, u'routing_key': u'celery', u'no_ack': False, u'alias': None, u'queue_arguments': None, u'binding_arguments': None, u'bindings': [], u'auto_delete': False}

(Names accord with the CELERY_DEFAULT_QUEUE settings)

website/celery.py contains the basics (imports skipped):

app = Celery('proj')
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

I would therefore expect tasks generated by a webserver running with the development settings, to end up only in the development_queue, and so on. However, I see tasks being processed by a different queue, or by all three, which is problematic.

Are my expectations wrong in that this would be a good way to separate these tasks? All documentation on routing is about routing different tasks to different queues, which I don't need. I need to route all tasks of a certain site (environment) to a certain queue. What can I do to separate these environments?

yeah i think you are correct.You can set default queue as different queue based on envirnment.So you can run consumer for each queues using `-Q queue_name` — itzMEonTV, Feb 13 '15 at 18:44
Well, that is what I was trying (see `--queues myserver` which is the same as `-Q myserver`), but as I said, it doesn't work. — Tino, Nov 03 '15 at 09:44
Have you had a look at the fanout_prefix and fanout_patterns options when using Redis as a broker? http://docs.celeryproject.org/en/latest/getting-started/brokers/redis.html#caveats — Carl, Nov 09 '15 at 20:11
@Carl I had not seen that option yet. It appears I can only set it to `True`. Do you know how it works? What "prefix" does it set? The docs say: "You have to set a transport option to prefix the messages so that they will only be received by the active virtual host", but what is the *active* virtual host? (What is a _virtual_ host?) Can't find it in the docs. — Tino, Nov 12 '15 at 09:11

score 4 · Accepted Answer · edited Jun 20 '20 at 09:12

I got an answer from the developer of Celery here: https://github.com/celery/celery/issues/2508, which is:

You have to set all of CELERY_DEFAULT_QUEUE, CELERY_DEFAULT_EXCHANGE and CELERY_DEFAULT_ROUTING_KEY. Otherwise you will end up with three queues all bound to the same exchange and routing key.

Or use the method here which will set it up explicitly: http://docs.celeryproject.org/en/latest/userguide/routing.html#changing-the-name-of-the-default-queue

That works!

score 2 · Answer 2 · answered Nov 11 '15 at 13:20

I have a similar set up to you but my solution is to use different RabbitMQ brokers for development, staging and production.

For example, in my personal development settings file I have:

CELERY_BROKER = "librabbitmq://user:user@my_machine.domain.com/"

In the production settings file I have:

CELERY_BROKER = "librabbitmq://prop:prod@main_cluster.domain.com/"

In my celery app module I have:

app = Celery('App', broker=settings.CELERY_BROKER)

I don't know whether this helps or not, but I saw your post on the celery-users board, and since you haven't had any other responses...

Yeah, that is what I ended up doing: using different redis databases for the broker. It works but it just felt that it should work with the DEFAULT_QUEUE setting ;-) — Tino, Nov 12 '15 at 09:06

Celery: how to separate different environments with different workers?

2 Answers2