4

Is there any way to delay a Celery task from running based on a condition? Before it moves from scheduled to active I would like to perform a quick check to see if my machine can run the task based on the arguments provided and my machine's state at the time. If it's not, it halts the scheduled queue and waits until the condition is satisfied.

I've looked around at the following points but it didn't seem to cut it:

  • Celery's Signals: closest thing I could get to is task_prerun() but regardless of what I put in there, the task will get run and it doesn't halt the other scheduled tasks from running. There's also worker_ready() but that doesn't look at the upcoming task's arguments to do the check.
  • Database Lock (also here as well): I can have each of the tasks start running normally and then do the check at the beginning of the task's run but if I set a periodic interval to check if condition is met, I lose the order of the active queue as condition can be met at any point and one of the many active tasks will be able to continue. This is where the database lock comes in and is so far the most feasible solution. I can make a lock every time I do the check and if the condition's not met, it stays locked. When the condition's finally met, I release the lock for the next item in the queue, preserving the queue's original order.

I find it surprising that celery doesn't have this functionality to specify if/when the next item in the scheduled queue is ready to be run.

boses
  • 400
  • 4
  • 12
  • Which condition do you want to use? Can you describe input data? Is it stored in db? – Danila Ganchar Mar 12 '17 at 18:36
  • I'm using Celery as a kind of cloud service with long heavy handed tasks. The input data is basically user's request to run a task in a simple python dictionary format. Ideally I would like to call a function before starting the task that would return true or false whether there's enough memory to run the virtual machine associated with the task, so that when memory is available, then we can start the task. For now I'm saving the order of the queue in a database and checking the memory after the task has started and before the virtual machine has launched. – boses Mar 13 '17 at 21:00
  • Could you provide show some code with user request, task, comments etc? – Danila Ganchar Mar 15 '17 at 15:07
  • 1
    After looking online again, I find it's a design issue. My tasks are asynchronous to a point that they can run in the background and do not share data. On the other hand, running all those tasks in the background can limit memory and get to the point where the rest of the items on the queue _should_ wait for a running task to complete before starting. The simple functionality I'm looking for is a conditional function in the broker that will return to keep a task on "reserved" until a condition is met, in this case, memory is available on a remote machine, and _then_ add it to the worker. – boses Apr 14 '17 at 15:09
  • I don't wanna use redis to replicate the queue as storing queues is what celery was designed for but that's what I've resorted to. I'm hoping to reverse engineer celery to find the exact code where the task is sitting on the queue and the broker is ready to send it to the worker, and from there add a function that blocks sending it to the worker until the conditional is met depending on the task. – boses Apr 14 '17 at 15:34
  • @boses you found a solution for this? – Himanshu Jain Oct 04 '18 at 17:02
  • @HimanshuJain Not that I remember, I've moved on from said project. – boses Oct 05 '18 at 18:06

0 Answers0