10

We are considering to use Kafka in our for messaging and our applications are developed using Spring. So, we have planned to use spring-kafka.

The producer puts the message as HashMap object into the queue. We have JSON serializer and we assumed that the map will be serialized and put into the queue. And here is the producer config. 

spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
        key-serializer: org.springframework.kafka.support.serializer.JsonSerializer
        value-serializer: org.springframework.kafka.support.serializer.JsonSerializer

On the other hand, we have a listener which listens to the same topic where the producer has published the message. Here is the consumer config:

spring:
   kafka:
       consumer:
            group-id: xyz
            key-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
            value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer

Our listener method:

  public void listener(SomeClass abx)

We were expecting the json will be de-serialized and an object of type "SomeClass" will be generated. But apparently, it throws de-serialization exception.

We saw few articles and the suggestion was to do something like:

 @Bean
  public ConsumerFactory<String, Car> consumerFactory() {
    return new DefaultKafkaConsumerFactory<>(consumerConfigs(), new StringDeserializer(),
        new JsonDeserializer<>(Car.class));
  }

We don't want to write some code for creating the Deserializer. Is there any boilerplate thing which we are missing? Any help will be appreciated!!

Thiru
  • 2,541
  • 4
  • 25
  • 39
  • 1
    Docs for serialization/deserialization: https://docs.spring.io/spring-kafka/reference/htmlsingle/#serdes Looks like it should work just with @KafkaListener annotation – Pospolita Nikita Jun 29 '18 at 07:53
  • They are suggesting to do something like new JsonDeserializer<>(Bar.class); which I dont want todo, as it will grow when we have more listeners with different types of object – Thiru Jun 29 '18 at 07:59
  • 2
    I don't think it's a problem to have a single configuration class for Kafka. it's just a part of app configuration. But I will watch this issue, cause it's interesting if the solution exists. – Pospolita Nikita Jun 29 '18 at 08:19
  • 1
    Sure. Thanks for suggestions!! – Thiru Jun 29 '18 at 08:21
  • You can set the required type using a property - see my answer. – Gary Russell Jun 29 '18 at 12:03
  • In case anyone comes here with the same question, I recommend reading answers in this question: https://stackoverflow.com/questions/55109508/spring-kafka-no-type-information-in-headers-and-no-default-type-provided – dmitryb Mar 25 '23 at 14:23

1 Answers1

20

See the boot documentation. In particular:

You can also configure the Spring Kafka JsonDeserializer as follows:

spring.kafka.consumer.value-deserializer=org.springframework.kafka.support.serializer.JsonDeserializer

spring.kafka.consumer.properties.spring.json.value.default.type=com.example.Invoice

spring.kafka.consumer.properties.spring.json.trusted.packages=com.example,org.acme

Community
  • 1
  • 1
Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • What if I had multiple types? for ex: Invoice, Payment, This and That etc. How does Spring know how to deserialize ? – RamPrakash Feb 26 '23 at 17:33
  • 2
    There are several options. You can override the properties at the listener level - see https://docs.spring.io/spring-kafka/docs/current/reference/html/#annotation-properties - just omit the `spring.kafka.consumer.properties` part. Or you can add a function - see https://docs.spring.io/spring-kafka/docs/current/reference/html/#serdes-type-methods Or, use a message converter instead and the framework will detect the type from the listener method signature - see https://docs.spring.io/spring-kafka/docs/current/reference/html/#messaging-message-conversion – Gary Russell Feb 26 '23 at 21:37