Spring-Kafka consumer's listener method is not called due to parse error

Question

I'm trying to implement a Kafka consumer using Kafka-Spring (spring-boot). I send a message using a producer but Spring throws an exception and my listener method is not called because of this exception.
Here's my listener method:

@Service
public class KafkaExchangeApi {

    @KafkaListener(topics = ORDER_TOPIC, groupId = "group_id")
    public void handleOrderResponse(@Payload GatewayOrder response) {
       System.println("order caught");
    }

}


@AllArgsConstructor
@NoArgsConstructor(force = true)
@Builder
@Getter
@ToString
public class GatewayOrder {

    private final long uid;
    
    private final long orderId;

    private final int userCookie;

    private final long size;
    private final OrderAction action;
    private final OrderType orderType;

    private final String symbol;

    // mutable fields

    private List<GatewayDeal> deals = new ArrayList<>();

    @Setter
    private BigDecimal price;

    @Setter
    private long filled;

    @Setter
    private GatewayOrderState state;

}

And this is my configuration:

server:
  port: 9000
spring:
  kafka:
    consumer:
      bootstrap-servers: 192.168.1.3:9092
      group-id: group-id
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
    producer:
      bootstrap-servers: 192.168.1.3:9092
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer

I thought that the JSON message sent by the producer is malformed and I tested to see if it can be parsed using the JSON parser and I confirmed that it can be parsed...
So, why do I get this exception?

020-09-10 17:04:41.878  INFO 5946 --- [ntainer#1-0-C-1] o.a.k.c.c.internals.SubscriptionState    : [Consumer clientId=consumer-group_id-2, groupId=group_id] Resetting offset for partition order-0 to offset 0.
2020-09-10 17:04:41.879  INFO 5946 --- [ntainer#1-0-C-1] o.s.k.l.KafkaMessageListenerContainer    : group_id: partitions assigned: [order-0]
2020-09-10 17:04:55.250 ERROR 5946 --- [ntainer#1-0-C-1] o.s.k.l.SeekToCurrentErrorHandler        : Backoff none exhausted for ConsumerRecord(topic = order, partition = 0, leaderEpoch = 0, offset = 0, CreateTime = 1599746694173, serialized key size = -1, serialized value size = 160, headers = RecordHeaders(headers = [], isReadOnly = false), key = null, value = {"uid":1, "orderId":1, "userCookie": 0, "size": 1, "action": "ASK", "orderType": "IOC", "symbol": "ABC", "deals":null, "price":1, "filled": 0, "state":"ACTIVE"})

org.springframework.kafka.listener.ListenerExecutionFailedException: Listener method could not be invoked with the incoming message
Endpoint handler details:
Method [public void  tradeapi.service.KafkaExchangeApi.handleOrderResponse( tradeapi.model.internal.GatewayOrder)]
Bean [ tradeapi.service.KafkaExchangeApi@2143ea5e]; nested exception is org.springframework.messaging.converter.MessageConversionException: Cannot handle message; nested exception is org.springframework.messaging.converter.MessageConversionException: Cannot convert from [java.lang.String] to [ tradeapi.model.internal.GatewayOrder] for GenericMessage [payload={"uid":1, "orderId":1, "userCookie": 0, "size": 1, "action": "ASK", "orderType": "IOC", "symbol": "ABC", "deals":null, "price":1, "filled": 0, "state":"ACTIVE"}, headers={kafka_offset=0, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@fb7548f, kafka_timestampType=CREATE_TIME, kafka_receivedPartitionId=0, kafka_receivedTopic=order, kafka_receivedTimestamp=1599746694173, kafka_groupId=group_id}], failedMessage=GenericMessage [payload={"uid":1, "orderId":1, "userCookie": 0, "size": 1, "action": "ASK", "orderType": "IOC", "symbol": "ABC", "deals":null, "price":1, "filled": 0, "state":"ACTIVE"}, headers={kafka_offset=0, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@fb7548f, kafka_timestampType=CREATE_TIME, kafka_receivedPartitionId=0, kafka_receivedTopic=order, kafka_receivedTimestamp=1599746694173, kafka_groupId=group_id}]; nested exception is org.springframework.messaging.converter.MessageConversionException: Cannot handle message; nested exception is org.springframework.messaging.converter.MessageConversionException: Cannot convert from [java.lang.String] to [ tradeapi.model.internal.GatewayOrder] for GenericMessage [payload={"uid":1, "orderId":1, "userCookie": 0, "size": 1, "action": "ASK", "orderType": "IOC", "symbol": "ABC", "deals":null, "price":1, "filled": 0, "state":"ACTIVE"}, headers={kafka_offset=0, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@fb7548f, kafka_timestampType=CREATE_TIME, kafka_receivedPartitionId=0, kafka_receivedTopic=order, kafka_receivedTimestamp=1599746694173, kafka_groupId=group_id}], failedMessage=GenericMessage [payload={"uid":1, "orderId":1, "userCookie": 0, "size": 1, "action": "ASK", "orderType": "IOC", "symbol": "ABC", "deals":null, "price":1, "filled": 0, "state":"ACTIVE"}, headers={kafka_offset=0, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@fb7548f, kafka_timestampType=CREATE_TIME, kafka_receivedPartitionId=0, kafka_receivedTopic=order, kafka_receivedTimestamp=1599746694173, kafka_groupId=group_id}]
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.decorateException(KafkaMessageListenerContainer.java:1925) ~[spring-kafka-2.5.5.RELEASE.jar:2.5.5.RELEASE]
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeErrorHandler(KafkaMessageListenerContainer.java:1913) ~[spring-kafka-2.5.5.RELEASE.jar:2.5.5.RELEASE]
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:1812) ~[spring-kafka-2.5.5.RELEASE.jar:2.5.5.RELEASE]
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeWithRecords(KafkaMessageListenerContainer.java:1739) ~[spring-kafka-2.5.5.RELEASE.jar:2.5.5.RELEASE]
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:1636) ~[spring-kafka-2.5.5.RELEASE.jar:2.5.5.RELEASE]
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:1366) ~[spring-kafka-2.5.5.RELEASE.jar:2.5.5.RELEASE]
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1082) ~[spring-kafka-2.5.5.RELEASE.jar:2.5.5.RELEASE]
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:990) ~[spring-kafka-2.5.5.RELEASE.jar:2.5.5.RELEASE]
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
    at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
Caused by: org.springframework.messaging.converter.MessageConversionException: Cannot handle message; nested exception is org.springframework.messaging.converter.MessageConversionException: Cannot convert from [java.lang.String] to [ tradeapi.model.internal.GatewayOrder] for GenericMessage [payload={"uid":1, "orderId":1, "userCookie": 0, "size": 1, "action": "ASK", "orderType": "IOC", "symbol": "ABC", "deals":null, "price":1, "filled": 0, "state":"ACTIVE"}, headers={kafka_offset=0, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@fb7548f, kafka_timestampType=CREATE_TIME, kafka_receivedPartitionId=0, kafka_receivedTopic=order, kafka_receivedTimestamp=1599746694173, kafka_groupId=group_id}], failedMessage=GenericMessage [payload={"uid":1, "orderId":1, "userCookie": 0, "size": 1, "action": "ASK", "orderType": "IOC", "symbol": "ABC", "deals":null, "price":1, "filled": 0, "state":"ACTIVE"}, headers={kafka_offset=0, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@fb7548f, kafka_timestampType=CREATE_TIME, kafka_receivedPartitionId=0, kafka_receivedTopic=order, kafka_receivedTimestamp=1599746694173, kafka_groupId=group_id}]
    at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:340) ~[spring-kafka-2.5.5.RELEASE.jar:2.5.5.RELEASE]
    at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:86) ~[spring-kafka-2.5.5.RELEASE.jar:2.5.5.RELEASE]
    at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:51) ~[spring-kafka-2.5.5.RELEASE.jar:2.5.5.RELEASE]
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:1880) ~[spring-kafka-2.5.5.RELEASE.jar:2.5.5.RELEASE]
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:1862) ~[spring-kafka-2.5.5.RELEASE.jar:2.5.5.RELEASE]
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:1799) ~[spring-kafka-2.5.5.RELEASE.jar:2.5.5.RELEASE]
    ... 8 common frames omitted
Caused by: org.springframework.messaging.converter.MessageConversionException: Cannot convert from [java.lang.String] to [ tradeapi.model.internal.GatewayOrder] for GenericMessage [payload={"uid":1, "orderId":1, "userCookie": 0, "size": 1, "action": "ASK", "orderType": "IOC", "symbol": "ABC", "deals":null, "price":1, "filled": 0, "state":"ACTIVE"}, headers={kafka_offset=0, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@fb7548f, kafka_timestampType=CREATE_TIME, kafka_receivedPartitionId=0, kafka_receivedTopic=order, kafka_receivedTimestamp=1599746694173, kafka_groupId=group_id}]
    at org.springframework.messaging.handler.annotation.support.PayloadMethodArgumentResolver.resolveArgument(PayloadMethodArgumentResolver.java:145) ~[spring-messaging-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at org.springframework.kafka.annotation.KafkaListenerAnnotationBeanPostProcessor$KafkaNullAwarePayloadArgumentResolver.resolveArgument(KafkaListenerAnnotationBeanPostProcessor.java:901) ~[spring-kafka-2.5.5.RELEASE.jar:2.5.5.RELEASE]
    at org.springframework.messaging.handler.invocation.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:117) ~[spring-messaging-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:148) ~[spring-messaging-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:116) ~[spring-messaging-5.2.8.RELEASE.jar:5.2.8.RELEASE]
    at org.springframework.kafka.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:48) ~[spring-kafka-2.5.5.RELEASE.jar:2.5.5.RELEASE]
    at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:329) ~[spring-kafka-2.5.5.RELEASE.jar:2.5.5.RELEASE]
    ... 13 common frames omitted

Can you post the message and the class GatewayOrder . Also post the configurations — Tushar Jajodia, Sep 10 '20 at 12:06
@TusharJajodia thanks. I have edited my question and added the info you told. — xyzt, Sep 10 '20 at 12:25

Gary Russell · Answer 1 · 2020-09-10T15:01:48.600

With the StringDeserializer you need to add a JsonMessageConverter @Bean to the application. See https://docs.spring.io/spring-kafka/docs/2.6.0/reference/html/#messaging-message-conversion - Boot will auto-configure it into the listener container factory.

It's more efficient to use a ByteArrayDeserializer to avoid an unnecessary byte[] to String conversion.

Or you can use a suitably configured JsonDeserializer to the kafka consumer configuration.

Either way, you will need to add Jackson to the class path too.

Spring-Kafka consumer's listener method is not called due to parse error

1 Answers1