0

I'm entirely new to kafka. I got to work with simple consumer, and it was ok. Now i've got work with balanced consumer, the problem is i have only a vague idea of kafka in general.

    topic = client.topics["topic_name"]
    consumer_a = topic.get_balanced_consumer(consumer_group=b"group_code", zookeeper_connect="localhost:2181")
    consumer_b = topic.get_balanced_consumer(consumer_group=b"group_code", zookeeper_connect="localhost:2181")

    for message in consumer_a:
        if message is not None:
            pool.apply_async(self._handle_message, (message,))

    for message in consumer_b:
        if message is not None:
            pool.apply_async(self._handle_message, (message,))

This is my initial idea of approach. However the loop is just staying in consumer_a. I'm not sure how to handle the loop exit. I thought of having the loops also handled my another async call, but i'm not sure if its the right way to go.

Can any one advise on the same or point to some better layman understandable docs? Or am i missing some other concept?

My requirement is to have both consumers being processed simultaneously. Thanks

edit If anyone can explain the concept of partitions it will be useful as well.

Vipin Mohan
  • 1,631
  • 1
  • 12
  • 22
  • What are you trying to achieve here? By _balanced consumers_ do you mean that you are trying to implement a couple of consumers in order to split the incoming messages and load the balance? – Giorgos Myrianthous Aug 13 '19 at 14:11
  • Yes, i'm expecting around a lot of records coming through. There are about 20 tables to watch with maybe hundreds + requests coming through each – Vipin Mohan Aug 13 '19 at 14:53
  • Are you flexible in terms of kafka libraries for python? Balanced consumers are provided out-of-the-box by `kafka-python` and `confluent-kafka` libraries. If you are free to choose any of these libraries I can share some examples. – Giorgos Myrianthous Aug 13 '19 at 15:19
  • No, i'm restricted to use pykafka, as there is some functionality based on that. i did suggest confluent-kafka as well, however it got rejected – Vipin Mohan Aug 13 '19 at 15:46

0 Answers0