I'm using confluent_kafka package in python to do event streaming locally. There are multiple stages in my program. The first stage sends output as a message to a topic designated to the next stage. Each subsequent stage does the same (retrieve message from the designated topic and send event to another topic).
The problem is, the second stages onward, I can only get messages lower than 5% of the time. For example, in the topic first stage sends to, I've checked using terminal that there are over 100 messages in that topic, but I only got 2 messages in the next stage. It's not entirely not working cuz I do get some of them, but I can't figure out where went wrong.
Any idea? Appreciate any help!
# Here is the main part of my producer:
producer = Producer({'bootstrap.servers': "localhost:9092"})
producer.produce("first_topic", key=event_key, value=json.dumps(event, indent=4, default= str))
producer.flush()
# Here is the main part of my consumer:
conf = {'bootstrap.servers': 'localhost:9092',
'group.id': 'a_group',
'auto.offset.reset': 'earliest'}
consumer = Consumer(conf)
consumer.subscribe(topic)
msg = consumer.poll(timeout=1.0)
if msg is None:
print("No message available.")
pass
if msg.error():
if msg.error().code() == KafkaError._PARTITION_EOF:
# End of partition event
sys.stderr.write('%% %s [%d] reached end at offset %d\n' %
(msg.topic(), msg.partition(), msg.offset()))
print("Error partition_eof")
elif msg.error():
raise KafkaException(msg.error())
else:
key = msg.key().decode("utf-8")
data = msg.value().decode("utf-8")
return data
