2

Imagine having a long running task with a specific set of args and kwargs. Is there any chance to revoke all running and pending tasks with the same args/kwargs before starting a new task as Im only interested in the result of the last added task. (The underlying data changes inbetween two calls)

I tried iterating the results of inspect.active(), inspect.registered() and inspect.scheduled() to get ALL tasks and then filter/revoke those with my args and kwargs in question.

But this was not reliable as the inspecting of all workers and searching for tasks took way too long.

Anyone can get me in the right direction?

Patrick
  • 331
  • 3
  • 18
MichelB
  • 41
  • 1
  • 5

1 Answers1

0

If inspecting the workers is to slow, it might be better to check via a secondary data storage, for example a key-value store like redis. You add an "lock" for an function, so you know you already started it.

When queueing task:

  • check if already exists in your store
    • yes: raise Exception
    • no: add relevant task info as item to your store.
  • add task to celery task queue

A worker will execute it

  • execute it
  • remove from storage

There is a redis based ready made implementation:

celery-once (permalink to current commit).

You have to specify it as a base of a task

from celery import Celery
from celery_once import QueueOnce

@celery.task(base=QueueOnce)
def sum(a, b):
    ...

You can also specify which parameters to take into account and some more settings, read the readme there for more details.

luckydonald
  • 5,976
  • 4
  • 38
  • 58