I'm new to Spring Cloud Stream and Kafka Streams. I trying to building new Kstreams with mapped keys of input streams using Spring Cloud Stream and Kafka (Streams) binder.
During the deserialization of the joined streams an error occurred (see log below). My understanding of the error: GenericAvroSerde are used also for the joined KStream since it set as bindings through spring cloud stream. If my unterstanding is correct, how to specify a different Serde for the joined KStream? If not was is the root cause of the issue?
Error logs:
org.apache.kafka.streams.errors.StreamsException: ClassCastException invoking processor: second-first-join-other-join. Do the Processor's input types match the deserialized types? Check the Serde setup and change the default Serdes in StreamConfig or provide correct Serdes via method parameters. Make sure the Processor can accept the deserialized input of type key: java.lang.String, and value: my.package.MyFirstTopicValue.
Note that although incorrect Serdes are a common cause of error, the cast exception might have another cause (in user code, for example). For example, if a processor wires in a store, but casts the generics incorrectly, a class cast exception could be raised during processing, but the cause would not be wrong Serdes.
at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:150) ~[kafka-streams-3.1.2.jar:na]
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forwardInternal(ProcessorContextImpl.java:253) ~[kafka-streams-3.1.2.jar:na]
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:232) ~[kafka-streams-3.1.2.jar:na]
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:191) ~[kafka-streams-3.1.2.jar:na]
at org.apache.kafka.streams.kstream.internals.KStreamJoinWindow$KStreamJoinWindowProcessor.process(KStreamJoinWindow.java:55) ~[kafka-streams-3.1.2.jar:na]
at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:146) ~[kafka-streams-3.1.2.jar:na]
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forwardInternal(ProcessorContextImpl.java:253) ~[kafka-streams-3.1.2.jar:na]
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:232) ~[kafka-streams-3.1.2.jar:na]
at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:191) ~[kafka-streams-3.1.2.jar:na]
at org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:84) ~[kafka-streams-3.1.2.jar:na]
at org.apache.kafka.streams.processor.internals.StreamTask.lambda$process$1(StreamTask.java:731) ~[kafka-streams-3.1.2.jar:na]
at org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:809) ~[kafka-streams-3.1.2.jar:na]
at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:731) ~[kafka-streams-3.1.2.jar:na]
at org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:1296) ~[kafka-streams-3.1.2.jar:na]
at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:784) ~[kafka-streams-3.1.2.jar:na]
at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:604) ~[kafka-streams-3.1.2.jar:na]
at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:576) ~[kafka-streams-3.1.2.jar:na]
Caused by: java.lang.ClassCastException: class java.util.LinkedHashMap cannot be cast to class my.package.MySecondTopicValue (java.util.LinkedHashMap is in module java.base of loader 'bootstrap'; my.package.MySecondTopicValue is in unnamed module of loader 'app')
at org.apache.kafka.streams.kstream.internals.AbstractStream.lambda$reverseJoinerWithKey$1(AbstractStream.java:106) ~[kafka-streams-3.1.2.jar:na]
at org.apache.kafka.streams.kstream.internals.KStreamKStreamJoin$KStreamKStreamJoinProcessor.process(KStreamKStreamJoin.java:177) ~[kafka-streams-3.1.2.jar:na]
at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:146) ~[kafka-streams-3.1.2.jar:na]
... 16 common frames omitted
2022-10-31 15:55:03.913 ERROR 25964 --- [-StreamThread-1] org.apache.kafka.streams.KafkaStreams : stream-client [...-processors-group-161db237-fe34-4ed8-a6b5-b57ed2f09a73] Encountered the following exception during processing and Kafka Streams opted to SHUTDOWN_CLIENT. The streams client is going to shut down now.
org.apache.kafka.streams.errors.StreamsException: ClassCastException invoking processor: second-first-join-other-join. Do the Processor's input types match the deserialized types? Check the Serde setup and change the default Serdes in StreamConfig or provide correct Serdes via method parameters. Make sure the Processor can accept the deserialized input of type key: java.lang.String, and value: my.package.MyFirstTopicValue.
Note that although incorrect Serdes are a common cause of error, the cast exception might have another cause (in user code, for example). For example, if a processor wires in a store, but casts the generics incorrectly, a class cast exception could be raised during processing, but the cause would not be wrong Serdes.
Caused by: java.lang.ClassCastException: class java.util.LinkedHashMap cannot be cast to class my.package.MySecondTopicValue (java.util.LinkedHashMap is in module java.base of loader 'bootstrap'; my.package.MySecondTopicValue is in unnamed module of loader 'app')
application.properties snippet:
...
spring.cloud.stream.kafka.streams.bindings.myEventProcessor-in-0.consumer.keySerde=io.confluent.kafka.streams.serdes.avro.GenericAvroSerde
spring.cloud.stream.kafka.streams.bindings.myEventProcessor-in-0.consumer.valueSerde=io.confluent.kafka.streams.serdes.avro.GenericAvroSerde
spring.cloud.stream.kafka.streams.bindings.myEventProcessor-in-1.consumer.keySerde=io.confluent.kafka.streams.serdes.avro.GenericAvroSerde
spring.cloud.stream.kafka.streams.bindings.myEventProcessor-in-1.consumer.valueSerde=io.confluent.kafka.streams.serdes.avro.GenericAvroSerde
...
@Bean
public BiConsumer<KStream<GenericData.Record, GenericData.Record>, KStream<GenericData.Record, GenericData.Record>> myEventProcessor() {
return (mySecondGenericKStream, myFirstGenericKStream) -> {
// Building my first typed KStream from AvroGenericRecord
KStream<String, MyFirstTopicValue> myFirstWithStringKeyKStream = myFirstGenericKStream
.map((key, value) -> {
CdcRecordAdapter adapter = new CdcRecordAdapter(key, value);
MyFirstTopicValue myFirstTopicValue = adapter.getValueAs(MyFirstTopicValue.class);
String stringKey = myFirstTopicValue.getTransactionId();
return KeyValue.pair(stringKey, myFirstTopicValue);
})
.repartition(Repartitioned.with(new Serdes.StringSerde(), newFirstJsonSerde()).withName("myFirstWithStringKeyKStream"))
.peek((key, value) -> printOnConsole("myFirstWithStringKeyKStream", key, value.toString()));
// Building my second typed KStream from AvroGenericRecord
KStream<String, MySecondTopicValue> mySecondWithStringKeyKStream = mySecondGenericKStream
.map((key, value) -> {
CdcRecordAdapter adapter = new CdcRecordAdapter(key, value);
MySecondTopicKey mySecondTopicKey = adapter.getKeyAs(MySecondTopicKey.class);
MySecondTopicValue mySecondTopicValue = adapter.getValueAs(MySecondTopicValue.class);
String stringKey = mySecondTopicKey.getTransactionId();
return KeyValue.pair(stringKey, mySecondTopicValue);
})
.repartition(Repartitioned.with(new Serdes.StringSerde(), newSecondJsonSerde()).withName("mySecondWithStringKeyKStream"))
.peek((key, value) -> printOnConsole("mySecondWithStringKeyKStream", key, value.toString()));
// Join two mapped KStreams with new keys
mySecondWithStringKeyKStream
.filter((key, value) -> value.getCrudOperation() == CrudOperation.INSERT)
.leftJoin(
myFirstWithStringKeyKStream,
(readOnlyKey, mySecondValue, myFirstValue) -> new MySecondFirstBuilder(readOnlyKey, mySecondValue, myFirstValue).build(),
JoinWindows.ofTimeDifferenceAndGrace(Duration.ofSeconds(15L), Duration.ofSeconds(5)),
StreamJoined.with(Serdes.String(), newSecondJsonSerde(), newFirstJsonSerde())
.withName("second-first-topics-joined")
.withStoreName("second-firs-topics-joined"))
.peek((key, value) -> printOnConsole("second-first-topics-joined", key, value.toString()));
//.process(() -> new MyProcessor(myEventPublisher))
};
}
