`for .. in KafkaConsumer`: how to survive `eof`?

Question

for consumed_record in KafkaConsumer(config.kafka_topic,                                      
                                     bootstrap_servers=config.kafka_bootstrap_servers,
                                     group_id=config.kafka_group_id):
        __process_kafka_record(consumed_record)

The code above works fine except when EOF is seen. Then for loop terminates, however that's not the behaviour I'd like. I want my consumer to run endlessly unless the entire process dies. Conceptually I need something like if is_eof(): continue. What's the right way to do that with python-kafka package?

score 1 · Accepted Answer · answered Mar 02 '20 at 10:04

It looks like you're using kafka-python.

The correct way to consume records is to wrap that loop inside another loop that only stops if you want to.

The kafka-python project contains a few examples, like:

consumer = KafkaConsumer(bootstrap_servers='localhost:9092')
consumer.subscribe(['my-topic'])

while not self.stop_event.is_set():
    for message in consumer:
        print(message)
        if self.stop_event.is_set():
            break

consumer.close()

`for .. in KafkaConsumer`: how to survive `eof`?

1 Answers1