Kombu Consumer as Thrift Service

Question

I need to build a Kombu consumer which can be controlled programatically. All the examples I've seen are just trivial ones that tell you to use ctrl-c to stop the program.

My main application is running as a Twisted Thrift service and I'm thinking that I can somehow use Twisted reactor to deal with the eventloop inside my consumer but I can't figure out how.

Here is my consumer class. The start_consuming() part is fine except that it's blocking and I cannot call stop_consuming() from the outside.

from kombu import BrokerConnection, Exchange, eventloop, Queue, Consumer


class DMS():
    __routing_key = None
    __is_consuming = None
    __message_counter = 0

    def __init__(self, routing_key):
        print 'server: __init__()'
        self.__routing_key = routing_key

    def __handle_message(self, body, message):
        self.__message_counter += 1

        # Print count every 10,000 messsages.
        if (self.__message_counter % 10000) == 0:
            print self.__message_counter

    def start_consuming(self):
        print 'server: start_consuming()'
        self.__is_consuming = True
        exchange = Exchange('raven-exchange', type='topic', durable=False)
        queue = Queue(self.__routing_key, exchange, routing_key=self.__routing_key)

        with BrokerConnection('amqp://guest:guest@10.1.1.121:5672//') as connection:
            with Consumer(connection, queue, callbacks=[self.__handle_message]) as consumer:
                for _ in eventloop(connection):

                    if self.__is_consuming:
                        pass
                    else:
                        break

                consumer.cancel()
            connection.close()

    def stop_consuming(self):
        print 'server: stop_consuming()'
        self.__is_consuming = False

JensG · Answer 1 · 2014-06-21T01:38:34.520

The recommended way to route Thrift service calls through a MQ system is via oneway calls, because that is the most natural way to communicate via MQ and MessageBus systems.

struct Foo {
  1: string whoa
  2: i32 counter
}

service Whatever {
    oneway void FooBar(1: Foo someData, 2:i32 moreData)
}

A oneway call is a special form of a Thrift RPC call: As the name suggests, the call goes only in one direction. Neither return values nor exceptions (which are actually return values) are used with oneway. The call does send only the input args and does not wait for any values in return.

In order to establish a bi-di communication, the client needs to implement a similar service, designed to receive incoming answer messages. There are some samples in the Thrift /contrib folder, featuring 0MQ, Rebus and Stomp. Although they do not specifically deal with Python, the main idea should become clear.

Kombu Consumer as Thrift Service

1 Answers1