Celery Task Priority

Question

I want to manage tasks using Celery. I want to have a single task queue (with concurrency 1) and be able to push tasks onto the queue with different priorities such that higher priority tasks will preempt the others.

I am adding three tasks to a queue like so:

add_tasks.py

from tasks import example_task

example_task.apply_async((1), priority=1)
example_task.apply_async((2), priority=3)
example_task.apply_async((3), priority=2)

I have the following configuration:

tasks.py

from __future__ import absolute_import, unicode_literals
from celery import Celery
from kombu import Queue, Exchange
import time

app = Celery('tasks', backend='rpc://', broker='pyamqp://')

app.conf.task_queues = [Queue('celery', Exchange('celery'), routing_key='celery', queue_arguments={'x-max-priority': 10})]


@app.task
def example_task(task_num):
    time.sleep(3)
    print('Started {}'.format(task_num)
    return True

I expect the second task I added to run before the third, because it has higher priority but it doesn't. They run in the order added.

I am following the docs and thought I had configured the app correctly.

Am I doing something wrong or am I misunderstanding the priority feature?

score 11 · Accepted Answer · edited May 23 '17 at 16:04

11

There is a possibility that the queue has no chance to prioritize the messages (because they get downloaded before the sorting happens). Try with these two settings (adapt to your project as needed):

CELERY_ACKS_LATE = True
CELERYD_PREFETCH_MULTIPLIER = 1

Prefetch multiplier is 4 by default.

I had developed a sample application to realize Celery's priority tasking on a very small scale. Please have a look at it here. While developing it, I had encountered a very similar problem and this change in settings actually solved it.

Note that you also require RabbitMQ version 3.5.0 or higher.

edited May 23 '17 at 16:04

EngineerCamp

661
1
6
16

answered May 16 '17 at 10:04

Vijeth Aradhya

268
2
11

Thanks for your answer. I downloaded your project and ran it exactly as documented with the settings as you describe but still don't see the priority behaviour working. – EngineerCamp May 16 '17 at 23:28
@EngineerCamp I re-checked after cloning it again and it seemed to be working fine. There is a small preview video [here](https://www.youtube.com/watch?v=nQXO2kjGV9M&feature=youtu.be) with only 1 Celery worker so that it's easy to observe the tasks getting sorted. In the beginning of the video (_while the video is getting transcoded to 360p_), **observe how 720p gets more priority when 480p and 720p are clicked even when 480p is clicked first**. – Vijeth Aradhya May 18 '17 at 00:54
I gave it another go but still no luck. I am using Python 3 and am on Ubuntu 14.04 so maybe there are some differences there. – EngineerCamp May 19 '17 at 17:45
2

@EngineerCamp My RabbitMQ version is "3.5.7". _Please check on the [RabbitMQ website](https://www.rabbitmq.com/priority.html) and on [Celery FAQ page](http://docs.celeryproject.org/en/latest/faq.html#does-celery-support-task-priorities), and you will notice that you need 3.5.0 or higher version for priority queue implementation._ If that is not the case, then please let me know :) – Vijeth Aradhya May 20 '17 at 18:19
1

That was it, it turned out I had a very old version. I've upgraded it works perfectly. Thanks for the help :) – EngineerCamp May 23 '17 at 15:07
what about celery version? if I have new RabbitMQ version, can I use the priority with Celery 3.1? – ItayB Jul 04 '17 at 06:01

score 0 · Answer 2 · answered Jun 20 '23 at 06:51

0

If you are using celery above 6.0 use these:

        celery_instance.conf.task_acks_late = True
        celery_instance.conf.worker_prefetch_multiplier = 1

answered Jun 20 '23 at 06:51

Celery Task Priority

2 Answers2

Linked