3

I have a program that generates a large list of products through Python's itertools; basically a big set of various combinations of words. There are multiple sets to run through the products() function. What I would like to do is get Celery to help with this by giving each set to a different Celery Task, then combining them in the end. My understanding is that chords is the way to do this.

So essentially I have this:

callback = tabulate_results.subtask()
header = []
for combo in combos_to_run:
    header.append(run_product.subtask(args=(object_terms, combo)))
result = chord(header)(callback)
result.get()

And its two supporting, stripped-down functions:

from celery import subtask, chord
@task()
def run_product(object_list, combo_set):
    results = []
    for result in product(object_list, *combo_set):
        results.append(result)
    return results

@task()
def tabulate_results(result_sets):
    master_set = []
    for result_set in result_sets:
        master_set.extend(result_set)

    return master_set

At first, the chord task was showing up in celeryev, but I had a problem referenced here: Django Celery - Missing something but I have no idea what? Have results but can't get them about Celery returning an error having to do with result tracking through MySQL. I was indeed using MySQL for my result backend, and switching it to Redis got rid of it. However, now I have a new problem. When I run through code through the Django shell, no task appears in celeryev, and nothing is returned beyond this:

R IS: <GroupResult: 9f658e8d-591f-4fa9-9e79-4db0c51e8331 [9b199d1e-061f-413c-9521-4a3051dd121a, 2effbfb5-c9dc-4569-a63f-656c233a9387, 80911a60-6a22-46bb-83a1-d5a84c659794, 70acfa43-8ffe-4bc8-8ff1-1df6def035e1, dd417423-d1f6-44eb-8c4b-2ded40d7614f, fbff8adc-815d-459c-b914-b30528dbbd39]>

Basically a Celery message, but with no data. The code also never returns, and my cursor is left hanging. When I control-C out, the line number is something in Celery that appears to be waiting. I have no idea for what though since celeryev gives me no task. I've confirmed that other tasks do show in celeryev.

I've tested my functions normally, without running them as Celery tasks, and they return normally.

SHORT VERSION I'm trying to get Celery chords to help with an intensive Python task in my Django app, but they don't appear to be returning any result or getting entered into celeryev. Redis backend. Celery version 3.0.15. Django 1.4.

Community
  • 1
  • 1
kensentor
  • 443
  • 2
  • 15
  • Have you tried setting a time limit for the header tasks, and defining the [error handling](http://docs.celeryproject.org/en/latest/userguide/canvas.html#error-handling)? – gdvalderrama Aug 23 '16 at 12:12

1 Answers1

2

First off, it's generally considered bad practice to have one task wait on the result of another (this is why one should used chained subtasks)-- source

Aside from that, the syntax you've pasted for your chord is incorrect. Here's the relevant link: http://docs.celeryproject.org/en/master/getting-started/next-steps.html#chords. Amongst other issues, your code does not pass any tasks to the chord. A chord is simply a group of tasks that get processed as a group, after which a callback is fired off. From the documentation, here's the correct syntax for a chord:

>>> from celery import chord
>>> from proj.tasks import add, xsum

>>> chord((add.s(i, i) for i in xrange(10)), xsum.s())().get()
90

Note that the chord takes two arguments, the group of initial tasks, on the one hand, and the callback (separated by a comma), on the other.

This is a start, but it sounds like spending more time with the documentation will help you out more than anything.

Benjamin White
  • 779
  • 6
  • 25
  • 1
    If you look at this [other example](http://docs.celeryproject.org/en/latest/userguide/canvas.html#chords) though, the format matches. – gdvalderrama Aug 23 '16 at 12:10