Celery: How to ignore task result in chord or chain?

Question

I'm using celery, I have several tasks which needed to be executed in order.

For example I have this task:

@celery.task
def tprint(word):
    print word

And I want to do something like this:

>>> chain(tprint.s('a') | tprint.s('b'))()

Then I get TypeError: tprint() takes exactly 1 argument (2 given).

The same with chord, in this situation which I need a task to be executed after a group of tasks:

>>> chord([tprint.s('a'), tprint.s('b')])(tprint.s('c'))

So how to deal with this situation? I don't care the result of each task, but they need to be executed in order.

Add a second parameter won't work:

@celery.task
def tprint(word, ignore=None):
    print word

>>> chain(tprint.s('a', 0) | tprint.s('b'))()

This will print out 'a' and 'None'.

celery chord header uses a group, which executes in parallel, so you shouldn't be using it if the order is relevant — gdvalderrama, Aug 23 '16 at 11:21

score 109 · Accepted Answer · answered Dec 09 '12 at 14:51

109

There is a built-in functionality to ignore result in chaining and others - immutable subtask. You can use .si() shortcut instead of .s() or .subtask(immutable=True)

More details here: http://docs.celeryproject.org/en/master/userguide/canvas.html#immutability

answered Dec 09 '12 at 14:51

Vlad Frolov

7,445
5
33
52

3

upvoted! i read the docs but still havent understood the terminology, what exactly is immutability supposed to do if you dont mind sharing – PirateApp May 30 '18 at 15:08
3

They seem to abuse the term, but the idea behind this option is simple. It marks the function as one that does not expect a result from a chained computation passed to its arguments. – Vlad Frolov May 30 '18 at 18:05
2

I now understand what `si` is used for, thanks! – Florent Oct 08 '22 at 13:56
2

That link points to a 404. The new location is: https://docs.celeryq.dev/en/stable/userguide/canvas.html#immutability – sc3000 Mar 06 '23 at 15:57

ygesher · Answer 2 · 2016-10-10T09:37:18.083

One possible solution has already posted, but I'd like to add further clarification and an alternate solution (and in some cases a superior one).

The error you're seeing, which indicates that your task's signature needs to take into account a second parameter, is due to the fact that when calling tasks in a chain, Celery automatically pushes each tasks result as the first parameter of the following task.

From the docs:

Tasks can be linked together, which in practice means adding a callback task:

>>> res = add.apply_async((2, 2), link=mul.s(16))
>>> res.get()
4

The linked task will be applied with the result of its parent task as the first argument

Therefore, in your case, you could rewrite your task like this:

@celery.task
def tprint(result, word):
    print word

~~If you're not going to do anything with the result, you may as well ignore it, by changing the decorator thus:~~

@celery.task(ignore_result=True)

And then you won't have to change your task signature.

Sorry, that last point needs further research.

score -1 · Answer 3 · answered Dec 03 '12 at 05:08

-1

You can try doing something like this. Instead of having a single parameter for function tprint you can have 2 parameters

def tprint(word, x=None):
    print word

then

chain(tprint.s('a', 0) | tprint.s('b'))()

answered Dec 03 '12 at 05:08

NitinJ

351
1
3
9

I'd suggest using `*ignore, **kwignore` arguments to cover any combination – Jesse the Game Dec 03 '12 at 05:12
1

This solution does not generalize. If he needs to chain 3 functions, would you add a 3rd parameter to also ignore? You are messing up the function's namespace and overall semantics. The .si() recommendation is the correct answer. – Chris Johnson Aug 27 '13 at 09:02

score -2 · Answer 4 · answered Dec 04 '12 at 09:56

Finally find a workaround for this, a chain decorator will do this job.

I don't know how exactly celery did it, but celery seems force bind previous task's result to next task's first argument.

So here's an example:

def chain_deco(func):

    @functools.wraps(func)
    def wrapper(chain=None, *args, **kwargs):
        if chain is False:
            return False

        func(*args, **kwargs)
        return True

    return wrapper


@celery.task
@chain_deco
def hello(word):
    print "hello %s" % word

Now this will give the right output.

>>> (hello.s(word='a') | hello.s(word='b'))()

OR

>>> (hello.s('a') | hello.s('b'))(True)

And decorator also provide the ability to stop a chain in the middle(make the later cascade fail.)

The same mechanism should work for chord too.

Celery: How to ignore task result in chord or chain?

4 Answers4