0

I'm using Kafka 2.2.0, so the option to configure max.compaction.lag.ms per topic is not yet available.

Globally I have log.cleaner.max.compaction.lag.ms=86400000 (one day):

For the topic __consumer_offsets, I'd like to delay the compaction to at least three days, to keep some short-term history.

I could set min.compaction.lag.ms to 259200000 (three days), but it seems contrary to the smaller log.cleaner.max.compaction.lag.ms. What would happen if I did so nonetheless?

Tobias Hermann
  • 9,936
  • 6
  • 61
  • 134

1 Answers1

1

The compaction will be performed if either happens:

  1. Both conditions for dirty/clean ratio and min.compaction.lag are fulfilled

or

  1. max.compaction.lag is fulfilled

So it is OK to have min.compaction.lag.ms bigger than max.compaction.lag.ms. Since they are used in 2 different conditions and do not contradict each other.

BTW, what did you choose in the end? What were your results?

Anatolii Stepaniuk
  • 2,585
  • 1
  • 18
  • 24
  • "So it is OK to have `min.compaction.lag.ms` less than `max.compaction.lag.ms`. -> Sure, but I was talking about the other way around, i.e., "max less than min". – Tobias Hermann Mar 15 '21 at 06:31
  • "what did you choose in the end? What were your results?" -> Nothing. I simply did not increase `min.compaction.lag.ms` for `__consumer_offsets` and abandoned the idea of having some short-term history for that topic. :-) – Tobias Hermann Mar 15 '21 at 06:31
  • 1
    @TobiasHermann Sorry, I've updated the answer based on your comment – Anatolii Stepaniuk Mar 16 '21 at 14:07