9

I'm using Django Celery with Redis to run a few tasks like this:

header = [
    tasks.invalidate_user.subtask(args = (user)),
    tasks.invalidate_details.subtask(args = (user))
]

callback = tasks.rebuild.subtask()

chord(header)(callback)

So basically the same as stated in documentation.

My problem is, that when this task chord is called, celery.chord_unlock task keeps retrying forever. Tasks in header finish successfully, but because of chord_unlock never being done, callback is never called.

Guessing that my problem is with not being able to detect that the tasks from header are finished, I turned to documentation to look how can this be customized. I've found a section, describing how the synchronization is implemented, there is an example provided, what I'm missing is how do I get that example function to be called (i.e. is there a signal for this?).

Further there's a note that this method is not used with Redis backend:

This is used by all result backends except Redis and Memcached, which increment a counter after each task in the header, then applying the callback when the counter exceeds the number of tasks in the set.

But also says, that Redis approach is better:

The Redis and Memcached approach is a much better solution

What approach is that? How is it implemented?

So, why is chord_unlock never done and how can I make it detect finished header tasks?

I'm using: Django 1.4, celery 2.5.3, django-celery 2.5.5, redis 2.4.12

Martin Tóth
  • 1,747
  • 3
  • 24
  • 35

3 Answers3

8

You don't have an example of your tasks, but I had the same problem and my solution might apply.

I had ignore_result=True on the tasks that I was adding to a chord, defined like so:

@task(ignore_result=True)

Apparently ignoring the result makes it so that the chord_unlock task doesn't know they're complete. After I removed ignore_result (even if the task only returns true) the chord called the callback properly.

Tisho
  • 8,320
  • 6
  • 44
  • 52
Chris Martin
  • 81
  • 1
  • 2
0

I had the same error, I changed the broker to rabbitmq and chord_unlock is working until my task finishes (2-3 minutes tasks)

when using redis the task finishes and chord_unlock only retried like 8-10 times every 1s, so callback was not executing correctly.

[2012-08-24 16:31:05,804: INFO/MainProcess] Task celery.chord_unlock[5a46e8ac-de40-484f-8dc1-7cf01693df7a] retry: Retry in 1s [2012-08-24 16:31:06,817: INFO/MainProcess] Got task from broker: celery.chord_unlock[5a46e8ac-de40-484f-8dc1-7cf01693df7a] eta:[2012-08-24 16:31:07.815719-05:00]

... just like 8-10 times....

changing broker worked for me, now I am testing @Chris solution and my callback function never receives the results from the header subtasks :S, so, it does not works for me.

celery==3.0.6

django==1.4

django-celery==3.0.6

redis==2.6

broker: redis-2.4.16 on Mac OS X

panchicore
  • 11,451
  • 12
  • 74
  • 100
0

This could cause a problem such that; From the documentation;

Note:

If you are using chords with the Redis result backend and also overriding the Task.after_return() method, you need to make sure to call the super method or else the chord callback will not be applied.

def after_return(self, *args, **kwargs):
    do_something()
    super(MyTask, self).after_return(*args, **kwargs)

As my understanding, If you have overwritten after_return function in your task, it must be removed or at least calling super one.

Bottom of the topic:http://celery.readthedocs.org/en/latest/userguide/canvas.html#important-notes

Ahmet DAL
  • 4,445
  • 9
  • 47
  • 71