Multiple consumers with same group id in Python

Question

Does anybody know how to run multiple consumers with same group id in Python? I've tried following

a = Consumer({'bootstrap.servers': 'localhost:9092', 'group.id': 'dd1',
              'default.topic.config': {'auto.offset.reset': 'smallest'}})
b = Consumer({'bootstrap.servers': 'localhost:9092', 'group.id': 'dd1',
              'default.topic.config': {'auto.offset.reset': 'smallest'}})
c = Consumer({'bootstrap.servers': 'localhost:9092', 'group.id': 'dd1',
              'default.topic.config': {'auto.offset.reset': 'smallest'}})
a.subscribe([topic_to_read])
b.subscribe([topic_to_read])
c.subscribe([topic_to_read]) 
running = True
while running:
    msg1 = a.poll(timeout=timeout)
    msg2 = b.poll(timeout=timeout)
    msg3 = c.poll(timeout=timeout)

But this is not working. So I've tried using multiprocessing lib but I am not able to make it work.

why do you say it's not working. What are you expecting to be the behaviour putting them in the same group? How many partitions do you have in topic_to_read? — groo, Oct 22 '17 at 09:07

score 1 · Answer 1 · answered Oct 24 '17 at 02:48

1

A group ID is a unique ID to each consumer. If you are subscribing to a same topic with multiple consumers, you must have different group ID's or else only one of those consumers will get the messages.

answered Oct 24 '17 at 02:48

bmarkham

1,590
15
27

score 0 · Answer 2 · answered Oct 24 '17 at 06:37

Check how many partitions you have for that topic. The number of consumers within a group id should not exceed the number of partitions of the topic the group is consuming from. Else the extra consumers will remain idle. ALSO CHECK IF YOU ARE ASSIGNING DIFFERENT CLIENID/CONSUMERID FOR EACH OF THE CONSUMERS.

Multiple consumers with same group id in Python

2 Answers2