Is there an example of how to create a GlobalKTable to keep count from a KStream using Spring Cloud stream and using Functional approach?
Asked
Active
Viewed 616 times
0
-
Did you take a look here? https://github.com/spring-cloud/spring-cloud-stream-samples/blob/master/kafka-streams-samples/kafka-streams-global-table-join/src/main/java/kafka/streams/globalktable/join/KafkaStreamsGlobalKTableJoin.java – sobychacko Mar 25 '20 at 15:42
-
I looked at that but it doesn't seem to be counting...i.e. basically, the usecase is to count but to count I should know the previous count, not sure how to get the previous count – jka Mar 25 '20 at 15:55
2 Answers
0
Is implementing processor interface the right approach?
@Bean
public Consumer<KStream<String, Long>> processorsample() {
return input -> input.process(() -> new Processor<String, Long>() {
@Override
public void init(ProcessorContext context) {
if (state == null) {
state = (KeyValueStore<String, Long>) context.getStateStore("mystate");
}
}
@Override
public void process(String key, Long value) {
if (state != null) {
if (key != null) {
Long currentCount = state.get(key);
if (currentCount == null) {
state.put(key, value);
} else {
state.put(key, currentCount + value);
}
}
}
}
@Override
public void close() {
if (state != null) {
state.close();
}
}
}, "mystate");
}
jka
- 77
- 2
- 8
0
According to the documentation GlobalKTables are read-only, you cannot modify a global table during the processing.
Since GlobalKTables are consumers of a Kafka topic, you can just send your data to the GlobalKTable's source topic, and eventually, it's going to be added to the table. But you cannot be sure that the GlobalKTable will be updated immediately.
Simon
- 1
- 2