spring cloud stream kafka KTable as input not working

Question

Spring Cloud Stream Kafka, KTable as input not working

Sink.java

public interface EventSink {
    @Input("inputTable")
    KTable<?, ?> inputTable();
}

MessageReceiver.java

@EnableBinding(EventSink .class)
public class MessageReceiver {

    @StreamListener
    public void process(@Input("inputTable") KTable<String, Event> KTable) {

        // below code is just for representation. I need to do lot of things after getting this KTable
        KTable.toStream()
                .foreach((key, value) -> System.out.println(value));
    }
}

application.yml

server:
  port: 8083

spring:
  cloud:
    stream:
      kafka:
        streams:
          binder:
            application-id: kafka-stream-demo
            configuration:
              default:
                key:
                  serde: org.apache.kafka.common.serialization.Serdes$StringSerde
                value:
                  serde: org.springframework.kafka.support.serializer.JsonSerde
          bindings:
            inputTable:
              materialized-as: event_store
        binder:
          brokers: localhost:9092
      bindings:
        inputTable:
          destination: nscevent
          group: nsceventGroup

I'm getting below error

Exception in thread "kafka-stream-demo-1e64cf93-de19-4185-bee4-8fc882275010-StreamThread-1" org.apache.kafka.streams.errors.StreamsException: Deserialization exception handler is set to fail upon a deserialization error. If you would rather have the streaming pipeline continue after a deserialization error, please set the default.deserialization.exception.handler appropriately.
    at org.apache.kafka.streams.processor.internals.RecordDeserializer.deserialize(RecordDeserializer.java:80)
    at org.apache.kafka.streams.processor.internals.RecordQueue.addRawRecords(RecordQueue.java:97)
    at org.apache.kafka.streams.processor.internals.PartitionGroup.addRawRecords(PartitionGroup.java:117)
    at org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:677)
    at org.apache.kafka.streams.processor.internals.StreamThread.addRecordsToTasks(StreamThread.java:943)
    at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:831)
    at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:767)
    at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:736)
Caused by: java.lang.IllegalStateException: No type information in headers and no default type provided
    at org.springframework.util.Assert.state(Assert.java:73)
    at org.springframework.kafka.support.serializer.JsonDeserializer.deserialize(JsonDeserializer.java:370)
    at org.apache.kafka.streams.processor.internals.SourceNode.deserializeValue(SourceNode.java:63)
    at org.apache.kafka.streams.processor.internals.RecordDeserializer.deserialize(RecordDeserializer.java:66)
    ... 7 more

Can somebody please advise what is the issue? With KStream as input it is working, but not as KTable. Thanks in advance

score 0 · Answer 1 · answered Nov 13 '19 at 08:46

0

KTable is always converted using the native Serde feature of Kafka Streams. Framework level conversion is not done on KTable (although there is an issue out there to add it). Since you are using a custom type for value, you need to specify a proper Serde instead of using the default String serde. You can add these to the configuration.

spring.cloud.stream.kafka.streams.binder.configuration:
  default.value.serde: org.springframework.kafka.support.serializer.JsonSerde
  spring.json.value.default.type: RawAccounting

KTable don't auto convert as input channel

answered Nov 13 '19 at 08:46

Pratap A.K

4,337
11
42
79

1

Looks like you answered your own question. As you found out, `KTable` is always converted using native Serdes. However, when you move to the 3.0 version of Spring Cloud Stream, this is going to be the default rather than done by the framework. 3.0 makes it much cleaner how the binder infers the right Serdes to use. I suggest you look into the docs for 3.0. – sobychacko Nov 13 '19 at 15:25

spring cloud stream kafka KTable as input not working

1 Answers1