2

I'm kind of experimenting with the low level processor API. I'm doing data aggregation on incoming records using the processor API and writing the aggregated records to RocksDB.

However, I want to retain the records added in the rocksdb to be active only for 24hr period. After 24hr period the record should be deleted. This can be done by changing the ttl settings. However, there is not much documentation where I can get some help on this.

how do I change the ttl value? What java api should I use to set the ttl time to 24 hrs and whats the current default ttl settings time?

Barro
  • 335
  • 3
  • 16

1 Answers1

3

I believe this is not currently exposed via the api or configuration. RocksDBStore passes a hard-coded TTL when opening a RocksDB: https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java#L158 and the hardcoded value is simply TTL_SECONDS = TTL_NOT_USED (-1) (see line 79 in that same file).

There are currently 2 open ticket regarding exposing TTL support in the state stores: KAFKA-4212 and KAFKA-4273: https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20text%20~%20%22rocksdb%20ttl%22 I suggest you comment on one of them describing your use case to get them moving forward.

In the interim, if you need the TTL functionality right now, state stores are pluggable, and the RocksDBStore sources readily available, so you can fork it and set your TTL value (or, like the pull request associated with KAFKA-4273 proposes, source it from the configs).

I know this is not ideal and sincerely hope someone comes up with a more satisfactory answer.

Michal Borowiecki
  • 4,244
  • 1
  • 11
  • 18