3

A Celery task queue to calculate result of (2 + 2) - 3.

@app.task()
def add(**kwargs):
    time.sleep(5)
    x, y = kwargs['add'][0], kwargs['add'][1]
    return x + y

@app.task()
def sub(**kwargs):
    time.sleep(5)
    x = args[0]
    y = kwargs['sub'][0]
    return x - y

Sample task data = kwargs = {'add' : (2, 2), 'sub' : (3)}
Chaining the tasks: result = (add.s() | sub.s()).apply_async(kwargs = kwargs)

As per design, apply_async only applies the kwargs to the first task in the chain. What do I need to change to achieve the desired outcome?

Gh0sT
  • 317
  • 5
  • 16
  • 1
    The linked task will be applied with the result of its parent task as the first argument, so you can modify the `return` to include `kwargs` – danleyb2 Nov 26 '19 at 04:25
  • In reality, the tasks will perform very different things. For eg - task1 - Open browser to `url`. task2 - send `data` to specific `input` fields. You can see that its not a good design to return stuff from the task1. – Gh0sT Nov 26 '19 at 04:37
  • Can anyone help please? – Gh0sT Nov 26 '19 at 17:49

2 Answers2

3

So as of Celery v4.4.0rc4 there is no better way to do this other than passing kwargs to each task's signature. Although it does look like Ask Solem (Celery dev) is open to a feature request..

This is how the chain should look like: 

result = (add.s(job_data = job_data)| sub.s(job_data = job_data)).apply_async()

However since our chains have 10+ tasks, I had to come up with an easier way to write this. 

# Workflow generator
def workflow_generator(task_list, job_data):
    _tasks = tuple(getattr(task, 's')(job_data = job_data) for task in task_list)
    return chain(*_tasks).apply_async()

taskList = [add, sub]
job_data = {'add' : (2, 2), 'sub' : (3)}
result = workflow_generator(taskList, job_data)
Gh0sT
  • 317
  • 5
  • 16
0

Why not just partially bind the tasks before chaining them?

result = (add.s(add=(2,2)) | sub.s(sub=(3,3))).apply_async()
2ps
  • 15,099
  • 2
  • 27
  • 47