4

In kafka there is a log.retention.ms configuration property that can be set at the Broker level and it can be overridden by retention.ms on topic level.

Imagine I have a topic in my Kafka cluster with default config - no explicitly set retention. When I created the topic, the Broker level log.retention.ms was set to 7 days. In the meantime I changed this setting to 30 days. Does this mean that my existing topic will use the configuration that existed when the topic was created (7 days) or does it now use the new Broker config of 30 days?

Since the not-explicitly-set config values at topic level are not shown when I do kafka-topics.sh --describe --topic foo I find it difficult to verify.

dreamcrash
  • 47,137
  • 25
  • 94
  • 117
moffeltje
  • 4,521
  • 4
  • 33
  • 57

1 Answers1

4

Does this mean that my existing topic will use the configuration that existed when the topic was created (7 days) or does it now use the new Broker config of 30 days?

Your topic will use the configuration settings when it was created, that is, 7 days.

Only newly created topics will be set with a retention of 30 days. Note that setting this at the broker level implies a "default" value for topics that don't specify the retention time, but modifying this at the broker level won't affect existing topics, there's no "on cascade" update mechanism here.

Short answer: No. The only action that can alter the retention time set for a topic when it was created is specifically changing this value, for example, by using kafka-topics.sh --alter command.

Arvind Kumar Avinash
  • 71,965
  • 6
  • 74
  • 110
aran
  • 10,978
  • 5
  • 39
  • 69
  • so if a topic has retention time of 3 days, and I alter the retention time to 7 days, will kafka start counting from the time the broker receives the command? it's not counting from when the message is stored in broker ? – Liu Aug 28 '23 at 16:30
  • @Liu your three last days will be stored and then deleted 4 days later (as the retention doesn't reset for the already received messages, it will take the stored timestamp and check if passes 7 days). – aran Sep 01 '23 at 12:53