What is the right way to invoke celery.task.chunks?

Question

I am working on a parallel-able tasks with Python3 and celery. I like to divide it into several chunks, so those cost of communications over the network will be saved. However, the celery document does not reveal enough details on how to call the result chunks. Tried different ways and they do not work as I expected. My code segment is as follows:

@app.task(name='pl.startregret')
def startregret(**kwargs): 
    items = list(zip(range(1000), range(1000)))
    chunk = regretclick.chunks(items, 10)
    print(chunk)
    for c in chunk:
        print(c)


@app.task(name='pl.regretclick')
def regretclick(x,y):
    print('got it.')
    return x + y

I read some codes and thought that chunk in my code should be a generator. However, the printout shows

[2014-10-15 13:12:15,930: WARNING/Worker-2] args
[2014-10-15 13:12:15,931: WARNING/Worker-2] subtask_type
[2014-10-15 13:12:15,931: WARNING/Worker-2] kwargs
[2014-10-15 13:12:15,931: WARNING/Worker-2] immutable
[2014-10-15 13:12:15,931: WARNING/Worker-2] options
[2014-10-15 13:12:15,931: WARNING/Worker-2] task

Any suggestions on the right way to call the chunks?

Thanks,

Update: I have read the source code and tried chunk(). Looks like the only issue now the default queue is used instead of the queue defined in celeryconfig.

score 8 · Accepted Answer · answered Oct 15 '14 at 20:16

8

Consider a simple add task like this.

@app.task()
def add(x, y):
    return x + y

Here is a simple way to invoke a chunk task.

res = add.chunks(zip(range(10), range(10)), 2)()

This chunks the given 10 tasks to 5 tasks of each size 2 and adds the chunked tasks to the default queue. If you want to route it to a different queue then you have to specify it while invoking the task.

res = add.chunks(zip(range(10), range(10)), 2).apply_async(queue='my_special_queue')

and then start a worker for this queue to consume tasks

worker -A your_app worker -l info -Q my_special_queue

answered Oct 15 '14 at 20:16

Chillar Anand

27,936
9
119
136

This should be the right way. However, the current stable version does not support customized queue. This feature is only available in the dev version 3.20a1. I gave up and used the default queue finally. – Hu Cao Oct 16 '14 at 14:15
We can setup any number of custom queues and custom workers in any version. – Chillar Anand Oct 16 '14 at 14:17

What is the right way to invoke celery.task.chunks?

1 Answers1