Replacing RocksDB with In-memory state store in Kafka Streams

Question

I'm using Kafka Streams 0.10.1.1 release.

the RocksDB implementation for state store can't handle our 50k/msg rate so I want to change the state store to be the in-memory one. This should be possible according to the docs: http://docs.confluent.io/3.1.0/streams/architecture.html#state

However, when I implement this:

val stateStore = Stores.create(stateStoreName).withStringKeys().withStringKeys().inMemory().build()

val procSuppl: KStreamAggregate = ... // I'll spare the implementation details

streamBuilder.addSource(
  "mysource",
  new StringDeserializer(),
  new StringDeserializer(),
  "input_topic"
).addProcessor("proc", procSuppl,  "mysource").addStateStore(stateStore, "proc")

I end up with this error in runtime:

Caused by: java.lang.ClassCastException: org.apache.kafka.streams.state.internals.MeteredKeyValueStore cannot be cast to org.apache.kafka.streams.state.internals.CachedStateStore
2017-01-23T13:19:11.830674020Z  at org.apache.kafka.streams.kstream.internals.KStreamAggregate$KStreamAggregateProcessor.init(KStreamAggregate.java:62)

The implementation of the above method is:

public void init(ProcessorContext context) {
        super.init(context);
        store = (KeyValueStore<K, T>) context.getStateStore(storeName);
        ((CachedStateStore) store).setFlushListener(new ForwardingCacheFlushListener<K, V>(context, sendOldValues));
    }

Why is it trying to cast the state store to a CachedStateStore instance? How can I implement a simple in-memory state store which should be possible according to docs?

Thanks

Btw: `KStreamAggregate` is an internal class and the usage is on your own risk. — Matthias J. Sax, Jan 25 '17 at 21:34

score 7 · Answer 1 · edited Aug 01 '19 at 18:29

In order to create an in-memory state store, one needs to create a store supplier (using Stores factory object):

val storeSupplier = Stores.inMemoryKeyValueStore("in-mem")

Then you need to use the store supplier when materializing a KTable:

val wordCounts =  builder
  .stream[String, String]("streams-plaintext-input")
  .flatMapValues(textLine => textLine.toLowerCase.split("\\W+"))
  .groupBy((_, word) => word)
  .count()(Materialized.as(storeSupplier))

Obtain the queryable store:

val qStore = streams.store(
  wordCounts.queryableStoreName,
  QueryableStoreTypes.keyValueStore[String, Long])

Replacing RocksDB with In-memory state store in Kafka Streams

1 Answers1