0

I'm missing events when reading from a Kafaka queue because the consumer is updating the offset without an explicit commit even when enable_auto_commit is disabled.

from kafka import KafkaClient, TopicPartition


topic_name = "my_topic"

consumer = KafkaConsumer(topic_name, group_id="me", enable_auto_commit=False)

for i, message in enumate(consumer):
    if i == 5:
        expected = message.offset
        print(expected)
        break
    else:
       consumer.commit()

for message in consumer:
    value = message.offset
    print(value)
    consumer.commit()
    break

produces

>>> 50078
>>> 50079

If I don't commit the read, shouldn't it be read again the next time I start consuming messages?

Gree Tree Python
  • 529
  • 1
  • 6
  • 22

1 Answers1

0

It looks like you're using the kafka-python, am I correct? If so, there's a GitHub issue that explains a workaround, although I don't see a solution from the maintainer. So, although I don't have a reason why you're encountering this, based on that issue, I can recommend you commit the message at least once.

Lucia Cerchie
  • 61
  • 3