Retention on Kafka Streams > 2.10

Question

I am joining to KStreams thus I need to set JoinWindows. The joined data has computed Timestamps from CustomTimestampExtractors. The Data could be out of order. I can set the retention with the deprecated JoinWindows.until(long)-method but since its deprecated I am looking for another solution.

I found the windowstore.changelog.additional.retention.ms-property am I supposed to use it ? This would lead to all state stores being retained that long.

PSchoe · Accepted Answer · 2020-10-15T11:36:04.600

0

As the Documentation states:

Use JoinWindows#grace(Duration) instead

IMHO the doc is a bit misleading in contrast to:

Set the window maintain duration (retention time) in milliseconds. This retention time is a guaranteed lower bound for how long a window will be maintained.

But be careful I wanted my StateStore to retain as long as possible thus I was using JoinWindows.until(Long.MAX_VALUE) before. Using JoinWindows.grace(Duration.ofMillis(Long.Max_VALUE)) does complain that grace period can't be negative. There for you have to use smaller values.

edited Oct 15 '20 at 11:36

answered Oct 15 '20 at 11:27

PSchoe

66
9

Seems to be an limitation with `Duration` type -- but if you would just set `Duration.ofYears(100L)` it should be effectively unlimited :) – Matthias J. Sax Feb 28 '21 at 19:52

Retention on Kafka Streams > 2.10

1 Answers1