0

In Kafka, I know that there is a topic, named __consumer_offsets, which is used to store the offsets of consumers. As my understanding, a consumer will do two kinds of things:

  1. fetch data from partition.
  2. write the offsets to the topic __consumer_offsets.

After reading this link: https://www.learningjournal.guru/courses/kafka/kafka-foundation-training/offset-management/, I knew that there were two kinds of offsets for a Kafka consumer: the current offset and the committed offset.

I've read many docs and blogs of Kafka, but it seems that all descriptions about offsets of consumer are about the committed offset. For example, enable.auto.commit=true will make the consumer write the committed offset to the topic __consumer_offsets automatically and the function commitSync() can manually write the committed offset to the topic __consumer_offsets.

Now I want to know how Kafka processes the current offset. Will it be write into the topic __consumer_offsets too? How? When?

Yves
  • 11,597
  • 17
  • 83
  • 180
  • 1
    This [answer](https://stackoverflow.com/questions/64775491/does-kafka-distinguish-between-consumed-offset-and-commited-offset/64776504#64776504) might help you to understand. Your client will keep up of the *consumed* offsets per partition and you can shift it around with APIs such as `seek`. Nothing about a *consumed* offset is written into an internal topic. – Michael Heil Dec 03 '20 at 14:19
  • 1
    And in summary, it's just retained in memory by the client code and used when fetching the next batch. That's why when a consumer goes there is no record of what it had consumed other than the committed offset hence at least once semantics - a consumer may process records and then fail before committing. – Chris Dec 03 '20 at 22:01

0 Answers0