1

I often clean up all current messages in a Kafka topic by updating retention.ms to 10. That makes all messages will be expired after 10ms. However, sometimes, the messages cannot be cleaned up by that way. I had to drop and re-create the topic in order to clean up all messages.

I'm not sure it's related to the issue or not, but it often happens after all consumers of that topic have been stopped working by some reason.

What could be the root cause for this?

mkrieger1
  • 19,194
  • 5
  • 54
  • 65
Tin Nguyen
  • 399
  • 1
  • 6
  • 16
  • 1
    This post on [cleaning topics through retention](https://stackoverflow.com/questions/63137084/data-still-remains-in-kafka-topic-even-after-retention-time-size/63137573#63137573) might help to solve your problem. – Michael Heil Dec 02 '20 at 14:31

1 Answers1

0

The retention.ms field is a minimum time for the log cleaner. The log cleaner only runs once every so often (the Kafka docs state 300000 ms), and only on closed log segments (default size of 1GB), so you may have to wait for it to run or need more data in the topic

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Simon Clark
  • 624
  • 3
  • 11
  • We changed the retention.ms to 10 and waited in over 5 mins, but there is nothing happen. I think in 5 mins, the log cleaner should be ran right? – Tin Nguyen Aug 23 '18 at 15:19