Celery Storing unrecoverable task failures for later resubmission

Question

I'm using the djkombu transport for my local development, but I will probably be using amqp (rabbit) in production.

I'd like to be able to iterate over failures of a particular type and resubmit. This would be in the case of something failing on a server or some edge case bug triggered by some new variation in data.

So I could be resubmitting jobs up to 12 hours later after some bug is fixed or a third party site is back up.

My question is: Is there a way to access old failed jobs via the result backend and simply resubmit them with the same params etc?

Tisho · Answer 1 · 2012-07-22T14:33:14.497

4

You can probably access old jobs using:

CELERY_RESULT_BACKEND = "database"

and in your code:

from djcelery.models import TaskMeta
task = TaskMeta.objects.filter(task_id='af3185c9-4174-4bca-0101-860ce6621234')[0]

but I'm not sure you can find the arguments that the task is being started with ... Maybe something with TaskState...

I've never used it this way. But you might want to consider the task.retry feature? An example from celery docs:

@task()
def task(*args):
    try:
        some_work()
    except SomeException, exc:
        # Retry in 24 hours.
        raise task.retry(*args, countdown=60 * 60 * 24, exc=exc)

edited Jul 22 '12 at 14:33

answered Jul 22 '12 at 14:23

Tisho

8,320
6
44
52

Unfortunately .retry isnt useful when I no longer have a reference to the specific task as args/kwargs aren't stored on the failed task. I have to have my own model storing this information. – dpn Jul 24 '12 at 06:40
I' not sure to understand. What do you mean you have no reference to specific task? If the task fails - then retry will start it with the same args/kwargs after the specified countdown period. Otherwise you'll need to persist the arguments per task, and to restart it manually.. – Tisho Jul 24 '12 at 06:50
> Otherwise you'll need to persist the arguments per task, and to restart it manually.. Yep this is what I need to do, since the server may have restarted in the mean time etc. EDIT: quotes aren't working as documented heh. – dpn Jul 24 '12 at 18:03

score 3 · Accepted Answer · edited Aug 12 '12 at 10:55

3

From IRC

<asksol> dpn`: task args and kwargs are not stored with the result

<asksol> dpn`: but you can create your own model and store it there (for example using the task_sent signal)

<asksol> we don't store anything when the task is sent, only send a message. but it's very easy to do yourself

This was what I was expecting, but hoped to avoid.

At least I have an answer now :)

edited Aug 12 '12 at 10:55

Vebjorn Ljosa

17,438
13
70
88

answered Jul 22 '12 at 19:17

dpn

592
3
9

Celery Storing unrecoverable task failures for later resubmission

2 Answers2

Linked