Retrieve queue length with Celery (RabbitMQ, Django)

Question

I'm using Celery in a django project, my broker is RabbitMQ, and I want to retrieve the length of the queues. I went through the code of Celery but did not find the tool to do that. I found this issue on stackoverflow (Check RabbitMQ queue size from client), but I don't find it satisfying.

Everything is setup in celery, so there should be some kind of magic method to retrieve what I want, without specifying a channel / connection.

Does anyone have any idea about this question ?

Thanks !

score 4 · Answer 1 · edited May 24 '17 at 02:29

4

Here is an example on how to read the queue length in rabbitMQ for a given queue:

def get_rabbitmq_queue_length(q):
    from pyrabbit.api import Client
    from pyrabbit.http import HTTPError

    count = 0

    try:
        cl = Client('localhost:15672', 'guest', 'guest')
        if cl.is_alive():
            count = cl.get_queue_depth('/', q)
    except HTTPError as e:
        print "Exception: Could not establish to rabbitmq http api: " + str(e) + " Check for port, proxy, username/pass configuration errors"
        raise

    return count

This is using pyrabbit as previously suggested by Philip

edited May 24 '17 at 02:29

maxkoryukov

4,205
5
33
54

answered Aug 30 '16 at 14:20

Paul in 't Hout

421
5
8

so no way through celery as is? – Frederick Nord Jan 16 '18 at 17:39
@FrederickNord, not that I am aware of. My understanding is that the workers ( Celery ) is only concerned with it's current task, or at most the task it pre-fetched (from RabbitMQ). It's unaware of what is in RabbitMQ and intentionally so. – Paul in 't Hout Feb 05 '18 at 18:56

score 3 · Answer 2 · answered Jul 26 '13 at 11:14

3

PyRabbit is probably what you are looking for, it is a Python interface to the RabbitMQ management interface API. It will allow you to query for queues and their current message counts.

answered Jul 26 '13 at 11:14

Philip Cristiano

904
5
11

score 1 · Answer 3 · edited May 23 '17 at 13:40

1

You can inspect the workers in celery by using inspect module. Here is the guide.

Also for RabbitMQ there are some command line command.

edited May 23 '17 at 13:40

maxkoryukov

4,205
5
33
54

answered Jul 25 '13 at 16:53

Aamir Rind

38,793
23
126
164

2

thanks for your answer. But I don't need commands, I need python classes or methods. Plus, the inspect method will inspect the workers, not the queues, am I right ? – user2619608 Jul 26 '13 at 08:39
`inspect` **can not** retrieve the length of pending queue as I know. May be `inspect reserved`, but in my experience it doesn't help too.. – maxkoryukov May 24 '17 at 18:26
Indeed, the official Celery documentation states that you inspect the queues with the rabbitmqctl command line tool. That is pretty inconvenient when RabbitMQ runs on a different machine. – Klaws Apr 10 '19 at 09:12

Retrieve queue length with Celery (RabbitMQ, Django)

3 Answers3

Linked