Python Kafka read message once only

Question

I am trying to read the earliest/oldest message which has not been read before in Kafka.

import json

from kafka import KafkaConsumer
from kafka import KafkaProducer


ORDER_KAFKA_TOPIC = "order_details"
ORDER_CONFIRMED_KAFKA_TOPIC = "order_confirmed"

consumer = KafkaConsumer(
    ORDER_KAFKA_TOPIC,
    bootstrap_servers="localhost:29092",
    auto_offset_reset = "earliest",
    enable_auto_commit=False,

)

print("Gonna start listening")
while True:
    for message in consumer:

        print("Ongoing transaction..")
        consumed_message = json.loads(message.value.decode())
        print(consumed_message)

I have set the parameters:

    auto_offset_reset = "earliest",
    enable_auto_commit=False,

As per the docs:

The enable_auto_commit option is set to False when creating the consumer. This means that the consumer will not automatically commit its offsets, so it will only receive messages that have not yet been committed by another consumer. This allows the consumer to read only unread messages from the topic.

However I am still getting messages that have been read.

score 0 · Answer 1 · answered Dec 10 '22 at 15:44

You need to specify a consumer group id for auto.offset.reset config to do anything.

Plus, "unread" is a misnomer. It should say uncommitted within the consumer group. As in, one group can read the messages, but that has no control whether another group sees the messages.

Records aren't deleted upon consumption. If that's the expected behavior you want, then either Kafka isn't the correct tool, or you'll need to track message uniqueness on your own.

Python Kafka read message once only

1 Answers1