Add a delay between subtasks of a group

Question

I am creating a Celery group to execute a bunch of processes in parallel. The following is the code -

@periodic_task(run_every=timedelta(minutes=1))
def schedule_pivots():
    today_list="A,B,C,D"
    g = group(get_pivots.s(l,'OPT') for l in today_list.split(','))
    g.set(countdown=100).delay()

'get_pivots' is the task called parallely. This task calls an external API which rate-limits if called within 500ms. Is there a way for me to introduce a delay between each execution of the task 'get_pivots' ?

did you try signature form where you are able to send parameters as well add.s(2, 2, rateLimit='10/m') ? — PirateApp, Apr 01 '18 at 13:01

score 0 · Accepted Answer · answered Apr 01 '18 at 13:05

Just tested this

@celery.task(rate_limit='10/m')
def add(x, y):
    return x + y

While calling this task from the python interpreter

>>> g = group(add.s(i, i + 1) for i in range(10))
>>> r = g()
>>> r.get()

The last part freezes till the everything is done, Hope that helps

Add a delay between subtasks of a group

1 Answers1