Add n tasks to celery queue and wait for the results

Question

I'd add several jobs to the celery queue and wait for the results. I have many ideas about how I would accomplish this using some type of shared storage (memcached, redis, database, etc.), but I think it was something Celery could handle automatically, but I can't find any resources online.

Code example

def do_tasks(b):
    for a in b:
        c.delay(a)

    return c.all_results_some_how()

score 41 · Accepted Answer · edited Sep 11 '18 at 01:57

41

For Celery >= 3.0, TaskSet is deprecated in favour of group.

from celery import group
from tasks import add

job = group([
             add.s(2, 2),
             add.s(4, 4),
             add.s(8, 8),
             add.s(16, 16),
             add.s(32, 32),
])

Start the group in the background:

result = job.apply_async()

Wait:

result.join()

edited Sep 11 '18 at 01:57

floer32

2,190
4
29
50

answered Apr 07 '16 at 03:17

laffuste

16,287
8
84
91

3

Information about `.s()` can be found [in the `celery.Signature` docs, here](http://docs.celeryproject.org/en/latest/reference/celery.html#celery.Signature) – floer32 Sep 11 '18 at 01:34

falsetru · Answer 2 · 2019-07-29T19:54:02.853

15

Task.delay returns AsyncResult. Use AsyncResult.get to get result of each task.

To do that you need to keep references to the tasks.

def do_tasks(b):
    tasks = []
    for a in b:
        tasks.append(c.delay(a))
    return [t.get() for t in tasks]

Or you can use ResultSet:

UPDATE: ResultSet is deprecated, please see @laffuste 's answer.

def do_tasks(b):
    rs = ResultSet([])
    for a in b:
        rs.add(c.delay(a))
    return rs.get()

edited Jul 29 '19 at 19:54

answered Nov 01 '14 at 07:24

falsetru

357,413
63
732
636

Worked like a charm save `ResultSet` requiring a list of results (or empty list) in it's constructor. I submitted an edit to the post to correct it. – Prydie Nov 02 '14 at 20:25
1

@Prydie, Thank you for your feedback and correction. – falsetru Nov 03 '14 at 03:09

score 2 · Answer 3 · answered Nov 03 '14 at 10:40

I have a hunch you are not really wanting the delay but the async feature of Celery.

I think you really want a TaskSet:

from celery.task.sets import TaskSet
from someapp.tasks import sometask

def do_tasks(b):
    job = TaskSet([sometask.subtask((a,)) for a in b])
    result = job.apply_async()
    # might want to handle result.successful() == False
    return result.join()

Add n tasks to celery queue and wait for the results

3 Answers3

Linked