0

I have to listen to a topic on Kafka and consume the messages being published on it. My piece of code is working fine but sometimes gets stuck in the try block as poll returns empty dictionary.

Please suggest a better way to poll the topic. TIA!

consumer = KafkaConsumer(**consumer_configs)
    consumer.subscribe(topics=[topic_name])
    while True:
        try:
            records = consumer.poll(10000, 500)
            for message in records.values():
                for msg in message:
                    print(msg.value)
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
WW_
  • 99
  • 10

1 Answers1

0

sometimes gets stuck in the try block as poll returns empty dictionary

That is expected behavior if you've reached the end of the topic. The while loop is still necessary to have the consumer block by polling and wait for new events.

You could do something like this within the loop, though

records = consumer.poll(10000, 500)
if records:
    for message in records.values():
        ....
else:
  print('waiting for data...')
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Yes this worked with time.sleep in else condition. I have added my business logic in if condition but if i do consumer.commit(), it comes out of the loop. – WW_ Oct 17 '22 at 07:38
  • Is it necessary to add commit() if we are polling continuously? will it read from same offsets again if we dont commit the offsets? – WW_ Oct 17 '22 at 07:39
  • It shouldn't stop the loop if you commit. You can also enable auto commits in the consumer properties... If you don't commit **and** the app restarts, then yes, it'll start from the last committed offsets, or `auto.offset.reset` property if no consumer group exists – OneCricketeer Oct 17 '22 at 13:20