How can I access the list of all uncommitted messages in a topic in kafka
Asked
Active
Viewed 1,777 times
1
-
1I don't understand what you mean by this. You use a consumer like you already have, based on previous questions, to get the records at those offsets. That's the only way – OneCricketeer Jan 17 '22 at 12:42
-
So, can I access all the messages in a topic with the consumer without committing? – Gülsen Keskin Jan 17 '22 at 12:43
-
1It's possible, sure. But maybe you can explain why you don't need/want to commit any consumed offset? – OneCricketeer Jan 17 '22 at 12:45
-
Because I want to commit when I read it again with another consumer. – Gülsen Keskin Jan 17 '22 at 12:48
1 Answers
1
To access uncommitted messages, first assumes you are committing. Otherwise, you're just consuming a topic.
The only way to get any records from Kafka is by using consumer api.
Commits aren't required (disable auto commit and don't explicitly call commit methods in code). However, if the app restarts for any reason, the auto.offset.reset property will always apply to any topic you're consuming, meaning you either skip everything or have to block your main code execution until this consumer reads everything from the beginning of the topic. One popular app that doesn't commit any offsets or create a consumer group is the Confluent Schema Registry. Kafka Streams changelog topics also do this.
If you want "another consumer" to read the entire topic, it needs to have a different group id. That's it
OneCricketeer
- 179,855
- 19
- 132
- 245