Kafka Streams Global Store - Adding change log topic

Question

I have a situation where i want to maintain my own global table structure. Essentially, Global Store is linked to a processor and i use some calculations to create key and then store it to key-value store.

Since global state store doesn’t have change log topic, it uses original topic as change log. And in case of state restore, it just loads the topic data to global table (which will be wrong) since we constructed our own key & value.

Topic Data

Key, value
1234,abcd
4567,defg

StateStore
Key, value
12ab, abcd
45de, defg

My requirement is to create a global store with my custom keys ( not the one which comes directly from topic). Any solution ?

The following link answers my question Kafka streams use cases for add global store

Use Kafka Streams to map your records into the format you're wanting, then create the state store, like normal — OneCricketeer, Feb 28 '20 at 14:35
https://stackoverflow.com/questions/59029964/kafka-streams-use-cases-for-add-global-store — SunilS, Feb 29 '20 at 14:40

score 1 · Answer 1 · answered Mar 10 '20 at 03:58

This is currently not supported until https://issues.apache.org/jira/browse/KAFKA-7663 and https://issues.apache.org/jira/browse/KAFKA-8037 are resolved.

Currently, a global Processor cannot change the data without breaking the application. You can only put unmodified data into the global state store.

As a workaround, you could do a preprocessing step that modifies the data, writes it back into a topic, and you consume this topic into your global state store.

Kafka Streams Global Store - Adding change log topic

1 Answers1