Consume from kafka without infinite loop

Question

I'm currently using the Confluent kafka python client to consume messages from a kafka topic and the code runs fine inside of a while True loop as shown in examples in the documentation. However I would like to set up a cron job that only consumes from the topic once a day. The idea is the job will check the topic in the morning, consume all the messages in the topic at that point in time and then stop. I tried achieving this in python like this:

msg = kafka_consumer.consume()
while msg:
  msg_val = msg.value().decode('utf-8')
  // do something with msg
  msg = kafka_consumer.consume()

The problem with this is it never ends up consuming anything. I guess the first line never gets a message on the first try. It only works with while True but I don't want this code to run infinitely, just until the last message at that point in time has been consumed.

score 0 · Accepted Answer · answered Aug 25 '20 at 17:48

0

You could check the offsets of the consumer group within the loop, then break the loop once you are within some threshold of the "end"

You might also want to play around with the max.poll.records consumer configuration to give more control over how many records you get back

answered Aug 25 '20 at 17:48

OneCricketeer

179,855
19
132
245

I had consumer.poll(1) and increasing the timeout from 1 to 10 did the trick, thanks! – Tatiana Frank Aug 26 '20 at 13:27
Do you think you could share the code? I'm inept at using kafka-python, but need to read messages and exit the loop as well. – Mustansir Godhrawala Jul 12 '22 at 14:34

Consume from kafka without infinite loop

1 Answers1