Celery - resubmit failed task chain after max retries exceeded

Question

I'm using celery with django. I'm having to provide an option for the user to inspect a failed task, make modifications to the failed task data if necessary and submit it again. I have seen this thread - Celery Storing unrecoverable task failures for later resubmission. So I understand that celery does not store the original args and kwargs of the task and we need to take care of that. I'm fine with doing that. But if I have a main task "MainTask1" that submits a chain "SubTask1 | SubTask2 | SubTask3" and if SubTask2 fails, then I see that SubTask3 wont be executed till SubTask2 succeeds. But if SubTask2 fails after max retries, then SubTask3 is never submitted.

My questions are -

1. When SubTask2 has failed, I can persist the args and kwargs of that. But how do I get information of the remaining tasks in the chain?

2. What exactly is stored in the columns 'result' and 'meta' of the table celery_taskmeta?

3. When is the table celery_tasksetmeta populated?

Thanks,

Answer 1

1. You could use results for that (e.g. AsyncResult(id).ready, see http://docs.celeryproject.org/en/latest/reference/celery.result.html), but this can not be done reliably with the amqp backend since only one process can retrieve the result.

2. The names used in the backend APi is inconsistent and outdated. This is because the terminology evolved quite a bit over time and the backends have been pretty much backwards compatible since celery 0.1. At first the tasks only stored data when it succeeded or failed. This used the fields status and result. Eventually I realized it was easy to extend this into supporting arbitrary states that updates as the task progresses, and this was simply implemented over what already existed.

   The result field contains the return value of the task when the task succeeded, or the exception raised in the task when it fails. Custom states can store arbitrary data in this field, so in newer terms a task state can have information attached to it and this is where that is stored.

   The meta field was added later because I was tired of adding new fields all the time (and forcing users to migrate the schema), it's used for new fields that does not have to be indexed and currently it only holds the children attribute, which is a list of subtask ids started by the task

   Hopefully it will be possible to clean this up some day. Most of the other 'growing-pains' has already been refactored away ages ago, but the result backend remains.

3. GroupResult.save (previously TaskSetResult) is used to store a group result for later retrieval, so you can get the list of task ids in a group just by keeping the group id. This feature is only used by the Redis result backend to implement chords at this point, this will change in the future, because it's not very efficient for groups with many tasks. The save option was implemented by user request, but I think it was a bad choice on my part to include it, and it'd be better for the user to do it (like the storing of task arguments/kwargs you mention).