Celery task that runs more tasks

Question

I am using celerybeat to kick off a primary task that kicks of a number of secondary tasks. I have both tasks written already.

Is there a way to easily do this? Does Celery allow for tasks to be run from within tasks?

My example:

@task
def compute(users=None):
    if users is None:
        users = User.objects.all()

    tasks = []
    for user in users:
        tasks.append(compute_for_user.subtask((user.id,)))

    job = TaskSet(tasks)
    job.apply_async() # raises a IOError: Socket closed

@task
def compute_for_user(user_id):
    #do some stuff

compute gets called from celerybeat, but causes an IOError when it tries to run apply_async. Any ideas?

Tasks and tasksets can be applied from within a task, but you should never wait for their results (see http://docs.celeryproject.org/en/latest/userguide/tasks.html#avoid-launching-synchronous-subtasks) — asksol, Jun 15 '11 at 11:55

score 38 · Accepted Answer · edited Aug 29 '16 at 23:24

To answer your opening questions: As of version 2.0, Celery provides an easy way to start tasks from other tasks. What you are calling "secondary tasks" are what it calls "subtasks". See the documentation for Sets of tasks, Subtasks and Callbacks, which @Paperino was kind enough to link to.

For version 3.0, Celery changed to using groups for this, and other, types of behavior.

Your code shows that you are already familiar with this interface. Your actual question seems to be, "Why am I getting a 'Socket Closed' IOError when I try to run my set of subtasks?" I don't think anyone can answer that, because you have not provided enough information about your program. Your excerpt cannot be run as-is, so we cannot examine the problem you're having for ourselves. Please post the stacktrace provided with the IOError, and with any luck, someone that can help you with your crasher will come along.

score 12 · Answer 2 · answered Oct 21 '13 at 07:57

12

You can use something like this (Support in 3.0 )

g = group(compute_for_user.s(user.id) for user in users)
g.apply_async()

answered Oct 21 '13 at 07:57

Abhilash Joseph

1,196
1
14
28

So in your implementation the "compute(users=None):" method is not required at all, yes? – Zoran Pavlovic Apr 21 '14 at 16:16
1

This is not recommended if you want to wait for the subtasks to complete and get back the result. – Siddharth Jun 16 '16 at 05:31

score 9 · Answer 3 · edited Dec 01 '16 at 22:49

9

And since version 3.0 'TaskSet' isn't the term anymore... Groups, Chains and Chords as a special type of subtask is the new thing, see http://docs.celeryproject.org/en/3.1/whatsnew-3.0.html#group-chord-chain-are-now-subtasks

edited Dec 01 '16 at 22:49

Community

1
1

answered Mar 05 '13 at 22:02

michel.iamit

5,788
9
55
74

Zheng Liu · Answer 4 · 2018-11-16T05:50:31.413

-1

For the IOError mentioned, although the information here is not sufficient to tell what caused it, my guess is that you tried to establish a connection inside the task function, so whenever a task is called, a new connection is built. If the task is to be called thousand times, there will be thousand connections. This will flood the system socket manager and the IOError is its complaint.

edited Nov 16 '18 at 05:50

answered Nov 16 '18 at 03:08

Zheng Liu

292
2
11

Celery task that runs more tasks

4 Answers4

Linked