Celery: Task Singleton?

Question

I have a task that I need to run asynchronously from the web page that triggered it. This task runs rather long, and as the web page could be getting a lot of these requests, I'd like celery to only run one instance of this task at a given time.

Is there any way I can do this in Celery natively? I'm tempted to create a database table that holds this state for all the tasks to communicate with, but it feels hacky.

Will the following URL help you? http://blog.abodit.com/2010/09/singleton-tasks-a-taskfactory-for-the-task-parallel-library-with-run-only-one-semantics/ — Arthur Greef, Jan 08 '13 at 19:44

score 2 · Answer 1 · answered Aug 11 '12 at 06:41

2

You probably can create a dedicated worker for that task configured with CELERYD_CONCURRENCY=1 then all tasks on that worker will run synchronously

answered Aug 11 '12 at 06:41

user1039098

141
1
5

2

You can also use memcached/redis or any database as explained here: http://docs.celeryproject.org/en/latest/tutorials/task-cookbook.html#ensuring-a-task-is-only-executed-one-at-a-time – asksol Aug 16 '12 at 17:12

shaihulud · Answer 2 · 2017-06-13T10:02:58.397

You can use memcache/redis for that. There is an example on the celery official site - http://docs.celeryproject.org/en/latest/tutorials/task-cookbook.html

And if you prefer redis (This is a Django realization, but you can also easily modify it for your needs):

from django.core.cache import cache
from celery.utils.log import get_task_logger


logger = get_task_logger(__name__)


class SingletonTask(Task):
    def __call__(self, *args, **kwargs):
        lock = cache.lock(self.name)

        if not lock.acquire(blocking=False):
            logger.info("{} failed to lock".format(self.name))
            return

        try:
            super(SingletonTask, self).__call__(*args, **kwargs)
        except Exception as e:
            lock.release()
            raise e
        lock.release()

And then use it as a base task:

@shared_task(base=SingletonTask)
def test_task():
    from time import sleep
    sleep(10)

This realization is nonblocking. If you want next task to wait for the previous task change blocking=False to blocking=True and add timeout

Where does `cache.lock` come from? Is that specific to a backend? I can't find it in the Django docs. — DylanYoung, Jul 21 '17 at 14:30
Yep, it is specific to redis backend. lock is a wrapper over [SETNX](https://redis.io/commands/setnx) command that is usually used for this case. I'm using django-redis as a Redis backend for Django. — shaihulud, Jul 21 '17 at 14:55
Cool! For those not using Redis, `cache.add` should work similarly (though atomicity may not be guaranteed depending on the backend). — DylanYoung, Aug 08 '17 at 16:50

Celery: Task Singleton?

2 Answers2