How to include Type MetaData for Deserialization in Spring Kafka

Question

I am doing deserialization at the Listener in Spring Kafka. But this assumes that the type information was included or sent by a Spring Kafka producer. In my case the Json is being sent across by the Debezium MySQLConnector and it does not add this meta data. So I would like to add it to the requests. I understand its placed in the request somewhere in the JsonSerializer, and I looked at the source code but could not figure out exactly how to use this to add meta data type during serialization generically to a request. In particular what field holds this type information? And is it the class name of the java object that was serialized? I dont think just setting a default serializer is going to work because I have multiple consumers listening on different topics as one would expect. Except for the simplest cases this is just not going to work to set one default as i have many consumers and types that I am deserializing to. So this answer is not going to work in my case Kafka - Deserializing the object in Consumer

Update tried using Method Types on deserializer but have another issue: Kafka Spring Deserialzer returnType static method never called

Answer 1

See

public abstract class AbstractJavaTypeMapper implements BeanClassLoaderAware {

    /**
     * Default header name for type information.
     */
    public static final String DEFAULT_CLASSID_FIELD_NAME = "__TypeId__";

    /**
     * Default header name for container object contents type information.
     */
    public static final String DEFAULT_CONTENT_CLASSID_FIELD_NAME = "__ContentTypeId__";

    /**
     * Default header name for map key type information.
     */
    public static final String DEFAULT_KEY_CLASSID_FIELD_NAME = "__KeyTypeId__";

    /**
     * Default header name for key type information.
     */
    public static final String KEY_DEFAULT_CLASSID_FIELD_NAME = "__Key_TypeId__";

    /**
     * Default header name for key container object contents type information.
     */
    public static final String KEY_DEFAULT_CONTENT_CLASSID_FIELD_NAME = "__Key_ContentTypeId__";

    /**
     * Default header name for key map key type information.
     */
    public static final String KEY_DEFAULT_KEY_CLASSID_FIELD_NAME = "__Key_KeyTypeId__";

2 sets of headers (keys and values).

TypeId is for simple classes

If TypeId is a container List<?>

ContentTypeId is the contained type.

If TypeId is a Map

Key_TypeId is the key type.

This allows you to reconstruct a Map<Foo, Bar>.

These headers can either contain fully qualified class names, or tokens that map to class names via the classIdMappings map.

However, since version 2.5, it would be easier to use the new

Using Methods to Determine Types.

That way, you can set your own headers and examine them in the method.

EDIT

Here is a simple example:

@SpringBootApplication
public class Gitter76Application {

    public static void main(String[] args) {
        SpringApplication.run(Gitter76Application.class, args);
    }

    @Bean
    public NewTopic topic() {
        return TopicBuilder.name("gitter76").partitions(1).replicas(1).build();
    }

    @KafkaListener(id = "gitter76", topics = "gitter76")
    public void listen(Foo in) {
        System.out.println(in);
    }

}

public class Foo {

    private String bar;

    public Foo() {
    }

    public Foo(String bar) {
        this.bar = bar;
    }

    public String getBar() {
        return this.bar;
    }

    public void setBar(String bar) {
        this.bar = bar;
    }

    @Override
    public String toString() {
        return "Foo [bar=" + this.bar + "]";
    }

}

spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.value-deserializer=org.springframework.kafka.support.serializer.JsonDeserializer

spring.kafka.consumer.properties.spring.json.trusted.packages=com.example.demo

$ kafkacat -P -b localhost:9092 -t gitter76 -H __TypeId__=com.example.demo.Foo
{"bar":"baz"}
^C

2020-08-08 09:32:10.034  INFO 58146 --- [ gitter76-0-C-1] o.s.k.l.KafkaMessageListenerContainer    : gitter76: partitions assigned: [gitter76-0]
Foo [bar=baz]