Chords of Chords Timeout

Question

Using celery, I have constructed a chord of chords:

from celery import chord
chord(task1, chord(task2, task3))

However, it often raises a timeout:

Chord '0f3dd024-8fe3-4b1b-ab9b-6081569c9738' raised:  
"TimeoutError('Operation timed out (3.0)',)"  
    Traceback (most recent call last):  
    File "python2.7/site-packages/celery/backends/base.py", line 568, in on_chord_part_return   
    StopIteration  
    culprit = next(deps._failed_join_report())

Is there someway to modify the timeout setting to allow for this celery design ?

Note that there are subtasks within each task, so a chain has not seemed to ensure all subtasks have completed before starting the next task .

Another constraint is that I cannot call get() to do this in two steps as these chords are already being constructed in a task.

that doesn't seem to be a timeout - from looking at the celery source it seems that one of the subtasks raised an exception that is not being handled properly. is your results backend defined properly and are you allowing exceptions to propagate? — scytale, Aug 12 '15 at 09:58
It looks like there is a timeout in base.py of: ret = j(timeout=3.0, propagate=propagate) — user2701815, Aug 13 '15 at 15:18

score 0 · Answer 1 · answered Dec 19 '16 at 08:36

A chord completion function will raise the TimeoutError exception also if some/all of its group tasks have set: ignore_result=True

When the completion function fires it can't find any results for the completed group tasks, therefore it assumes they have been cleared from the result backend due to a timeout.

It's hard to tell without seeing your code if this is the case. Anyway, make sure you don't ignore results for the group tasks.

Chords of Chords Timeout

1 Answers1