1

So if a Kafka Consumer has auto commit enabled with a commit interval of 5 sec, and we consume a batch every 1 second ...

... will the consumer process messages multiple times before the auto commit has been completed?

NorwegianClassic
  • 935
  • 8
  • 26

1 Answers1

2

If you marked the batch processed, it will commit to the partition in the cluster every 5 sec. The consumer issues the next batch poll to get the next set of messages, so it would start from the next offset. Unless there is a error occurred, it's not likely to get the same batch of msg again.

Join the Redpanda's community slack channel to get more help from the core engineers :)

Christina Lin
  • 44
  • 2
  • Thanks, so the consumer internally keeps track of what it has processed? It just askes for the next set based on what it just processed? – NorwegianClassic Jan 05 '23 at 09:01
  • Offsets are cached in memory (for faster access). And yes, it will ask for the next set from it's current marked process offset. – Christina Lin Jan 05 '23 at 17:19