How to use a custom Kafka state store in Spring Cloud Stream application with functional appraoch

Question

I've a spring cloud stream application with latest version and I'm using functional approach. The requirement is to receive a message, avoid duplicate messages (because the producer may publish duplicate messages), then transform the message and finally put the count in a state store. To fulfill the requirement of avoiding duplicate messages, I'm thinking of using an intermediate state store but I'm not able understand how to link the state store with current KStream.

Following the the sample code:

@Bean
public Function<KStream<String, String>, KStream<String, Long>> sampleProcessor() {
    return source -> source
            .mapValues(value -> value.toUpperCase())
            .groupBy((key, value) -> value)
            .count(Materialized.as("count-store"))
            .toStream();
}

Ex: If I publish following messages:

m1 = Hello
m2 = World
m3 = Hello

Then only m1 and m2 should be consumed.

Answer 1

I'm able to do it by creating a StoreBuilder bean:

@Bean
public StoreBuilder<KeyValueStore<String, String>> storeBuilder() {
    return Stores.keyValueStoreBuilder(
            Stores.persistentKeyValueStore("value-store"),
            Serdes.String(),
            Serdes.String()
    );
}

And then using this store in transform method of KStream as follows:

@Bean
public Function<KStream<String, String>, KStream<String, Long>> sampleProcessor() {
    return source -> source
            .transform(() -> new Transformer<String, String, KeyValue<String, String>>() {
                private KeyValueStore<String, String> valueStore;

                @Override
                public void init(ProcessorContext context) {
                    valueStore = context.getStateStore("value-store");
                }

                @Override
                public KeyValue<String, String> transform(String key, String value) {
                    if (isNull(valueStore.get(value))) {
                        valueStore.put(value, "CONST");
                        return KeyValue.pair(key, value);
                    }
                    return null;
                }

                @Override
                public void close() {
                }
            }, "value-store")
            .mapValues(value -> value.toUpperCase())
            .groupBy((key, value) -> value)
            .count(Materialized.as("count-store"))
            .toStream();
}