Celery + Redis backend: How to limit queue size?

Question

Is there a way to limit queue size when I run Celery with Redis backend?

Something like x-max-length in queue predeclare for rabbitmq

score 1 · Answer 1 · answered Oct 09 '19 at 14:13

1

As far as I know that is not possible with Redis as backend.

answered Oct 09 '19 at 14:13

DejanLekic

18,787
4
46
77

Seems to be true. – Maksym Polshcha Oct 09 '19 at 14:28

score 0 · Answer 2 · answered Oct 09 '19 at 13:49

0

I think you are looking for prefetch limits in celery. Check it out docs.

answered Oct 09 '19 at 13:49

Bharat Gera

800
1
4
13

The prefetch limit is a limit for the number of tasks a worker can reserve for itself. I need to limit my queues. – Maksym Polshcha Oct 09 '19 at 13:59
That's interesting! Limiting a queue for executing the task, I haven't come across anything like that before. Please do share if you find out the way for the same. – Bharat Gera Oct 09 '19 at 15:36

score 0 · Answer 3 · answered Oct 17 '22 at 13:47

This might be a bit hacky but you could try using a Redis lock when calling the task. That way if another process wants to call the task, it would have to wait for the Redis lock to be released (this would happen when either the task is done running or when the timeout is reached). This would prevent too many tasks being added to the queue

 with r.lock(some_lock_name, blocking_timeout=10):
    your_celery_task.delay()

Celery + Redis backend: How to limit queue size?

3 Answers3