1

I am new to Kafka. I am doing some experiment as to how to purge messages in a kafka topic. I found that if we set "retention.ms" property for a topic to some less time value lets say 1 second, then after 1 seconds the messages in the topic will be purged as per my understanding.

I ran 1 producer which produced few messages to topic and stopped it after some time. At the same time I ran a console consumer so it got the generated messages. I started another consumer console for the same topic after retention time is elapsed lets say after 1-2 minutes. But too my surprise I was able to get the messages on that topic. Started console consumer after 2 minutes again when I fifnnaly didnt see any messages in topic. It took almost 3-4 minutes for kafka to purge the messages. Is there any additional settings required at Kafka so that messages will be purged instantly ?

Rahul Vedpathak
  • 1,346
  • 3
  • 16
  • 30

1 Answers1

2

Setting retention.ms will not guarantee message will be deleted immediately from topic. Even though it will be marked for deletion.

If your message is in form of pair, then setting retention time is not good enough. You have to set following parameters also:

log.cleanup.policy
log.cleaner.min.compaction.lag.ms
log.cleaner.enable

Another set of parameters controls the deletion of message in case if they are present in your config:

log.retention.ms
log.roll.hours
Steephen
  • 14,645
  • 7
  • 40
  • 47