Celery task routing doesn't work when using current_app.send_task

Question

I have an issue with Celery queue routing when using current_app.send_task

I have two workers (each one for each queue)

python manage.py celery worker -E -Q priority --concurrency=8 --loglevel=DEBUG
python manage.py celery worker -Q low --concurrency=8 -E -B --loglevel=DEBUG

I have two queues defined in celeryconfig.py file:

# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from django.core.exceptions import ImproperlyConfigured

from celery import Celery
from django.conf import settings

try:
    app = Celery('proj', broker=getattr(settings, 'BROKER_URL', 'redis://'))
except ImproperlyConfigured:
    app = Celery('proj', broker='redis://')

app.conf.update(
    CELERY_TASK_SERIALIZER='json',
    CELERY_ACCEPT_CONTENT=['json'],
    CELERY_RESULT_SERIALIZER='json',
    CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend',
    CELERY_DEFAULT_EXCHANGE='tasks',
    CELERY_DEFAULT_EXCHANGE_TYPE='topic',
    CELERY_DEFAULT_ROUTING_KEY='task.priority',
    CELERY_QUEUES=(
        Queue('priority',routing_key='priority.#'),
        Queue('low', routing_key='low.#'),
    ),
    CELERY_DEFAULT_EXCHANGE='priority',
    CELERY_IMPORTS=('mymodule.tasks',)

CELERY_ENABLE_UTC = True
CELERY_TIMEZONE = 'UTC'

if __name__ == '__main__':
    app.start()

In the definition of tasks, we use decorator to explicit the queue:

@task(name='mymodule.mytask', routing_key='low.mytask', queue='low')
def mytask():
    # does something
    pass

This task is run indeed in the low queue when this task is run using:

from mymodule.tasks import mytask
mytask.delay()

But it's not the case when it's run using: (it's run in the default queue: "priority")

from celery import current_app
current_app.send_task('mymodule.mytask')

I wonder why this later way doesn't route the task to the "low" queue!

p.s: I use redis.

I can't find anything in the documents so I'm speculating here but it seems like the send_task acts as kind of a default fallback. If you need to use it I'm pretty sure you can pass it a keyword to pass it the routing key. — user2097159, Jan 15 '15 at 15:15
yes, we can pass the queue: current_app.send_task('mymodule.mytask', queue='low') but why pass it each time while it's defined in the task decorator? — mou55, Jan 15 '15 at 15:19

score 5 · Answer 1 · answered Jan 16 '15 at 11:02

send_task is a low-level method. It sends directly to the broker the task signature without going through your task decorator. With this method, you can even send a task without loading the task code/module.

To solve your problem, you can fetch the routing_key/queue from configuration directly:

  route = celery.amqp.routes[0].route_for_task("mymodule.mytask")
Out[10]: {'queue': 'low', 'routing_key': 'low.mytask'}
  celery.send_task("myodule.mytask", queue=route['queue'], routing_key=route['routing_key']`

Celery task routing doesn't work when using current_app.send_task

1 Answers1