0

What happens to data in a Kafka topic when retention.bytes limit is reached and cleanup.policy is set to compact? Which setting takes precedence? For example, if I create a topic by setting retention.bytes to 10000000 (10 MB) and cleanup.policy to "compact" and there enough unique keys to take 10MB disk space, what will happen to the data in Kafka topic when new messages are published to the topic?

Will Kafka retain the old unique keys and ignore retention.bytes config setting? Or will Kafka drop older messages? Or should I be setting the retention.bytes=-1 to make sure clean.policy=compact works as expected?

Shiva Prathipati
  • 26
  • 3

1 Answers1

0

Check Rohit Yadav's answer here.

There is no need to set log.retention to -1 or any other value. Your topics will be compacted and old messages never deleted (as per compaction rules).

If clean up policy is set to compact then a periodic cleanup occurs as explained in this post.

Shubham
  • 109
  • 14