I want to implement Kafka for messaging over a distributed microservice architecture.
I am using PyKafka and implemented dummy producer and (balanced)consumer. I assigned all consumers to the same consumer group. I have no problem using producers from Python and Console simultaneously, even runtime adding them.
However, I have a problem with consumers. I can create multiple Python consumers and even runtime add them. But when I add Console consumer (kafka-console-consumer) to the group with Python consumers, I am getting mutex error:
Error committing offsets for topic 'b'michal_sample_topic'' from consumer id 'b'Michals-MacBook-Pro.local:1722eea0-07d3-4be4-9d97-8b7fb15b0b30''(errors: {'pykafka.exceptions.UnknownMemberId': [0, 1]})
Moreover, both of these(even though they belong to the same consumer group) are consuming messages(Python consumers balance it among themselves and console consumers among themselves)
Now, I am new to Kafka, but my first impression was that Kafka should be agnostic to implementation of consumers, so combining them should be possible. Is issue in my understanding, PyKafka or my implementation of PyKafka?
Producer:
from pykafka import KafkaClient
from time import sleep
client = KafkaClient(hosts="localhost:9092")
print(client.brokers)
print(client.topics)
topic = client.topics[b'michal_sample_topic']
with topic.get_sync_producer() as producer:
while True:
producer.produce(
bytes(
input('Send test message:'),
'utf-8'
)
)
Consumer:
from pykafka import KafkaClient
client = KafkaClient(hosts="localhost:9092")
print(client.brokers)
print(client.topics)
topic = client.topics[b'michal_sample_topic']
balanced_consumer = topic.get_balanced_consumer(
consumer_group=b'testing',
auto_commit_enable=True,
zookeeper_connect='localhost:2181'
)
for message in balanced_consumer:
if message is not None:
print(f'{message.offset} {message.value}')
