celery - Tasks that need to run in priority

Question

In my website users can UPDATE they profile (manual) every time he want, or automatic once a day.

This task is being distributed with celery now.

But i have a "problem" :

Every day, in automatic update, a job put ALL users (+-6k users) on queue:

from celery import group
from tasks import *
import datetime
from lastActivityDate.models import UserActivity

today   = datetime.datetime.today()
one_day = datetime.timedelta(days=5)
today -= one_day

print datetime.datetime.today()

user_list = UserActivity.objects.filter(last_activity_date__gte=today)
g = group(update_user_profile.s(i.user.auth.username) for i in user_list)

print datetime.datetime.today()
print g(user_list.count()).get()

If someone try to do the manual update, they will enter on te queue and last forever to be executed.

Is there a way to set this manual task to run in a piority way? Or make a dedicated for each separated queue: manual and automatic?

you can limit the number of tasks executed, for example in hour. Then another tasks... Read their documentation... — catherine, Apr 04 '13 at 11:56

score 46 · Accepted Answer · edited Oct 15 '20 at 01:50

46

Celery does not support task priority. (v3.0)

http://docs.celeryproject.org/en/master/faq.html#does-celery-support-task-priorities

You may solve this problem by routing tasks.

http://docs.celeryproject.org/en/latest/userguide/routing.html

Prepare default and priority_high Queue.

from kombu import Queue
CELERY_DEFAULT_QUEUE = 'default'
CELERY_QUEUES = (
    Queue('default'),
    Queue('priority_high'),
)

Run two daemon.

user@x:/$ celery worker -Q priority_high
user@y:/$ celery worker -Q default,priority_high

And route task.

your_task.apply_async(args=['...'], queue='priority_high')

edited Oct 15 '20 at 01:50

zeraien

165
11

answered Apr 05 '13 at 06:16

Satoshi Yoshinaga

664
6
3

9

For anyone coming to this answer late (like me); it's vital to note that the two celery workers are running on different hosts - ie, two servers are consuming from the priority_high queue, and one server is consuming default – mafrosis Feb 01 '15 at 05:01
@Satoshi Yoshinaga can I also achieve queue specific concurrency by specificying "-c N" param in the daemon running commands? I need one of my queues to have only some workers, and other other a lot (heavy tasks queue). – bad_keypoints May 18 '15 at 05:58
2

Now you can also use [message priorities](http://docs.celeryproject.org/en/latest/whatsnew-3.0.html#redis-priority-support) – Chemary Oct 26 '16 at 17:47
10

Actual link - http://docs.celeryproject.org/en/latest/userguide/routing.html#routing-options-rabbitmq-priorities – pfctdayelise Mar 29 '17 at 04:04
4

to anyone who come across this answer. the Queue is from kombu package not queue package – Pitipong Guntawong Oct 06 '18 at 15:06
29

To those coming late to this answer, Celery now supports task priority to varying extents with RabbitMQ and Redis – Kevin Postlewaite May 03 '19 at 22:32
5

Why makes the high-priority queue high priority in the task routing setup example above? The fact that it is served by more workers than the default queue? The reason I'm asking is that I only have one host to run one or more celery workers, I want to prioritize some quick tasks and I'm not sure how to do it. – Alan Evangelista Feb 29 '20 at 00:10
@AlanEvangelista, in the above example, "priority_high" queue is so because tasks there can be executed in both workers whereas tasks in "default" can be only executed in the second worker. Thus, if the second worker is busy with default tasks that where queued before, still, the first worker can execute high priority tasks. – Pablo Guerrero Mar 28 '21 at 16:45

score 19 · Answer 2 · answered Aug 12 '21 at 08:08

19

If you use RabbitMQ transport then configure your queues the following way: settings.py

from kombu import Queue
...
CELERY_TASK_QUEUES = (
    Queue('default', routing_key='task_default.#', max_priority=10), 
    ...)

Then run your tasks:

my_low_prio_task.apply_async(args=(...), priority=1)
my_high_prio_task.apply_async(args=(...), priority=10)

Presently this code works for kombu==4.6.11, celery==4.4.6.

answered Aug 12 '21 at 08:08

Andrey St

388
3
10

3

Right answer. For anyone else wondering, 10 is the highest priority and 1 is the lowest. – Utkarsh Dalal Dec 02 '21 at 15:54
The docs say the priority values are 0 to 9 with 0 as the highest (and the default). And that the actual usable values are defined in the `priority_steps` setting which has a default of `[0, 3, 6, 9]`. But it seems `max_priority` changes that so in this example the priorities are 0 to 10, but that only works with RabbitMQ. It's a bit confusing. – Tim Tisdall Mar 25 '22 at 13:14
@TimTisdall which docs? – Gordon Wrigley Oct 19 '22 at 10:49
@GordonWrigley I found it again, but I think what I said is specific to only when Redis is the broker: [This means that even though there are 10 (0-9) priority levels, these are consolidated into 4 levels by default to save resources.](https://docs.celeryq.dev/en/stable/userguide/routing.html#redis-message-priorities) – Tim Tisdall Oct 19 '22 at 12:26
@TimTisdall and further down the same page "Note that priorities values are sorted in reverse when using the redis broker: 0 being highest priority." – Gordon Wrigley Oct 19 '22 at 14:52
1

@GordonWrigley - seems like this needs to be abstracted better from the particular broker used – Tim Tisdall Oct 19 '22 at 15:04
1

@TimTisdall I agree and documented better, especially since it's not the easiest to actually test, are my jobs being run? yes, in the right order?... – Gordon Wrigley Oct 20 '22 at 10:23

celery - Tasks that need to run in priority

2 Answers2