8

I am using celery 3 with Django.
I have a list of jobs in database. User can start a particular job which starts a celery task.
Now I want user to be able to start multiple jobs and it should add them to the celery queue and process them one after the other not in parallel as with async.

I am trying to create a job scheduler with celery where user can select the jobs to execute and they will be executed in sequential fashion.
If I use chain() then I cannot add new tasks to the chain dynamically.

What is ​the best solution?

John Moutafis
  • 22,254
  • 11
  • 68
  • 112
rohitnaidu19
  • 673
  • 5
  • 23
  • 1
    Possible duplicate of [Create celery tasks then run synchronously](http://stackoverflow.com/questions/26241381/create-celery-tasks-then-run-synchronously) – olofom May 16 '17 at 20:46
  • 1
    I don't think this is a duplicate of the linked question. The question is basically how make new jobs wait on previously scheduled jobs—something that is impossible in Celery. – drhagen Nov 06 '17 at 16:33

1 Answers1

2

A better primitive for you to use would be link instead of chain in Celery.

From the documentation:

s = add.s(2, 2)
s.link(mul.s(4))
s.link(log_result.s())

You can see how this allows you to dynamically add a task to be executed by iterating through the required tasks in a loop, and linking each one's signature. After the loop you would want to call something like s.apply_async(...) to execute them.

zarnoevic
  • 329
  • 3
  • 12