Kafka state stores forget old data?

Question

I try to query my local window state stores like this:

long timeFrom = 0;
long timeTo = System.currentTimeMillis();
WindowStoreIterator<Long> iterator = windowStore.fetch("topic name", timeFrom, timeTo);

I only get around about 36 hours of windowed data back, but would like to get far back in the past.

I tried to configure the retention in several ways, and also restarted my kafka server and zookeeper:

in config/server.properties: log.retention.hours=87600

bin/kafka-topics.sh --zookeeper zk.yoursite.com --alter --topic topicName --config retention.ms=86400000

Still Kafka does not keep old data.

score 1 · Accepted Answer · answered Sep 04 '17 at 23:42

1

I think you are looking for window retention, not log retention. If so, you can find out how to configure this to your liking here http://docs.confluent.io/current/streams/developer-guide.html#windowing

answered Sep 04 '17 at 23:42

dawsaw

funny thing, when defining this window retention, data from the past appear immediately - so they have never been deleted, but just not been fetched – micgn Sep 05 '17 at 19:23
yeah it's the window retention, so the data still exists in the change log to pull up, but you have to tell it to go get it :) – dawsaw Sep 05 '17 at 20:59

score 0 · Answer 2 · answered Sep 03 '17 at 03:34

0

check this setting log.retention.bytes to see if you're deleting data after a certain size. Retention policy can also be size based.

answered Sep 03 '17 at 03:34

moon

When you consume using consumer are you able to see the old data? – mrnakumar Sep 03 '17 at 21:36
the log.retention.bytes was not set at all and I guess it does not have a default. To be sure I set it now and will see what happens... – micgn Sep 04 '17 at 20:46
@mrnakumar I did not check yet. – micgn Sep 04 '17 at 20:47
Can you please check this? This will help us identify where the problem is, in Kafka or streams code itself – mrnakumar Sep 05 '17 at 06:49

2 Answers2