RabbitMQ/Pika blocking consumer with evenlet

Question

I am working on a larger application that requires eventlet, and now requires rabbitMQ as well. It appears that eventlet is causing the pika consumer thread to block execution of additional workers. I know Pika is not considered thread-safe, so I have everything, including the connection, within it's own thread. I would assume the blocking connection should only block the consumer thread. How can I get pika and eventlet to work together? In the below example, the workers thread never prints anything out, but commenting out eventlet.monkey_patch() allows both threads to execute.

import threading
import pika

import eventlet
eventlet.monkey_patch()


def callback(ch, method, properties, body):
    print body
    ch.basic_ack(delivery_tag=method.delivery_tag)


def consumer():
    connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
    channel = connection.channel()
    channel.queue_declare(queue='test', durable=True,
                          exclusive=False, auto_delete=False)
    channel.basic_consume(callback, queue='test')
    channel.start_consuming()


def start_consumer_thread():
    # initialize a listener thread
    consumer_thread = threading.Thread(target=consumer)
    consumer_thread.start()

def worker():
    start_consumer_thread()
    for x in range(1,10000):
        print x


x = threading.Thread(target=worker())
x.start()

score 2 · Accepted Answer · answered Jul 05 '18 at 14:14

2

Pika and eventlet.monkey_patch are not compatible. You will have to use eventlet without patching system calls, if that is possible.

_{NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.}

answered Jul 05 '18 at 14:14

Luke Bakken

8,993
2
20
33

Thanks. Is there a way to periodically check for a message in queue and consume it rather than have an never ending blocking loop? – user2242044 Jul 05 '18 at 17:55
Not sure if that is true. oslo.messaging is using pika, and it's monkey patching afaik. You should always monkey patch before importing other libraries. Try to move monkey_patch before the pika import. – eandersson Jul 12 '18 at 17:06
@eandersson - that doesn't appear to be the case anymore [link](https://github.com/openstack/oslo.messaging/blob/40532a2a457eba1179183654079e0ed81ddbcc48/releasenotes/notes/pika-driver-has-been-deprecated-e2407fa53c91fe5c.yaml). This is a shame since Pika is much more reliable than kombu or other python libraries for RabbitMQ. – Luke Bakken Jul 12 '18 at 17:44
Yea - it was a misstep that when they introduced pika support, they didn't allow for a mix between the pika and kombu drivers. So if you wanted to use pika, you had to take down your whole cloud to update the configuration. – eandersson Jul 12 '18 at 18:01
btw for reliability! I prefer my amqp library =] – eandersson Jul 12 '18 at 18:18

score 0 · Answer 2 · answered Aug 21 '18 at 03:13

I was able to get a pika consumer working with eventlet by monkey patching as early as possible, and also importing pika in a patched manner.

Import and patch stdlib first:

import eventlet
eventlet.monkey_patch()

Then import and patch pika itself:

pika = eventlet.import_patched('pika')

I used this import strategy in conjunction with the asynchronous_consumer_example: https://pika.readthedocs.io/en/stable/examples/asynchronous_consumer_example.html and used eventlet primitives rather than threading to achieve a non-blocking consumer.

RabbitMQ/Pika blocking consumer with evenlet

2 Answers2