1

I am running Kafka broker (version 2.12) with the default server setting on my Mac. I created a log compacted topic, with the following configuration -

Configs: cleanup.policy=compact,segment.bytes=1073741824,delete.retention.ms=100,segment.ms=100

I produced three records with same key/value. After 5 min (log.retention.check.interval.ms=300000), I read the topic from beginning. I see the following three records in the topic. I was expecting 1 record.

key1:aaa
key1:aaa
key1:aaa

After 10 min, I produced a record with same key, different value (key1:bbb). Then after 5 min, reading the topic from beginning results in following two records. I was expecting 1 record (key1:bbb).

key1:aaa
key1:bbb

Is there any additional Topic configuration I need to set? Any help or pointers would be highly appreciated.

Marios
  • 26,333
  • 8
  • 32
  • 52
Ahmed A
  • 3,362
  • 7
  • 39
  • 57
  • 1
    My answer in another post might answer your question: https://stackoverflow.com/questions/61430509/kafka-log-compaction-always-shows-two-last-records-of-same-key/61457387#61457387 – Michael Heil Aug 21 '20 at 21:04
  • @mikeThank you very much for providing the link, very good and detailed explanation. – Ahmed A Aug 21 '20 at 22:33
  • Glad it helped. If it answered your question you might consider upvoting my answer and closing this question as duplicate. Otherwise let us know if anything is still unclear. – Michael Heil Aug 22 '20 at 05:07

0 Answers0