Python to Spring Boot using Kafka: No type information in headers and no default type provided

Question

I'm trying to create a pure micro-service, where we are not constrained to specific to programming language, or framework.

Here I'm using Kafka to create the event pipeline.

Now I'm sending a message (event) from my python application:

producer = KafkaProducer(bootstrap_servers=broker)
# req is a dict {'a':1, 'c':-2}
json_message = json.dumps(req).encode('utf-8')
print("--------------------------")
print(json_message)
print("--------------------------")
headers = [
     ('content-type', 'application/json'),
     ('type', 'com.mua.cloud.testm.models.events'),
]
producer.send(topic+"-m", json_message,headers=headers)

And I'm just trying to consume it, in my Spring Boot application:

@Service
@Log4j2
public class TestHandler {

   @KafkaListener(topics = Constants.TOPIC_PREFIX + "-" + "python-trigger-m")
   @SneakyThrows
   public void listener(@Header(KafkaHeaders.CORRELATION_ID) byte[] corrId, TestEvent event) {
       System.out.println(corrId);
       System.out.println(event);
       System.out.println("----------------");
   }
}

And I've modified my configuration for this:

spring:
 …
 kafka:
   …
   consumer:
     value-deserializer: org.apache.kafka.common.serialization.ByteArrayDeserializer
     value-serializer: org.springframework.kafka.support.serializer.JsonSerializer

Registered a Bean (ref: https://stackoverflow.com/a/68749773/10305444):

@Configuration
public class JsonMessageConverterConfig {
   @Bean
   public JsonMessageConverter jsonMessageConverter() {
       return new ByteArrayJsonMessageConverter();
   }
}

But I keep getting this error on a loop:

TestMicro  ERROR    2023-07-18 19:08:08,922   [consumer-0-C-1] [o.s.k.l.KafkaMessageListenerContainer] - Message:  Consumer exception 
java.lang.IllegalStateException: This error handler cannot process 'SerializationException's directly; please consider configuring an 'ErrorHandlingDeserializer' in the value and/or key deserializer
    at org.springframework.kafka.listener.DefaultErrorHandler.handleOtherException(DefaultErrorHandler.java:151)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.handleConsumerException(KafkaMessageListenerContainer.java:1815)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1303)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
    at java.base/java.lang.Thread.run(Thread.java:1589)
Caused by: org.apache.kafka.common.errors.RecordDeserializationException: Error deserializing key/value for partition cloud-python-trigger-m-0 at offset 0. If needed, please seek past the record to continue consumption.
    at org.apache.kafka.clients.consumer.internals.Fetcher.parseRecord(Fetcher.java:1448)
    at org.apache.kafka.clients.consumer.internals.Fetcher.access$3400(Fetcher.java:135)
    at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.fetchRecords(Fetcher.java:1671)
    at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.access$1900(Fetcher.java:1507)
    at org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:733)
    at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:684)
    at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1277)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1238)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1211)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollConsumer(KafkaMessageListenerContainer.java:1531)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doPoll(KafkaMessageListenerContainer.java:1521)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1345)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1257)
    ... 3 common frames omitted
Caused by: java.lang.IllegalStateException: No type information in headers and no default type provided
    at org.springframework.util.Assert.state(Assert.java:76)
    at org.springframework.kafka.support.serializer.JsonDeserializer.deserialize(JsonDeserializer.java:583)
    at org.apache.kafka.clients.consumer.internals.Fetcher.parseRecord(Fetcher.java:1439)
    ... 15 common frames omitted

Also tried adding this configuration: spring.kafka.producer.properties.spring.json.add.type.headers=false

How can I resolve this issue?

NB:

I have tested the other way around, and I can send message from Spring Boot to Python using Kafka.
There are other already implemented functionalities, so it's kind of hard to add any config (as it may break existing codebase)

Approach 2:

I tried changing the config:


consumer:
 value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
 value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer

And the Kafka configuration like this:

@Bean
public Map<String, Object> consumerConfigs() {
   var config = new HashMap<String, Object>();
   …
   config.put(JsonDeserializer.TYPE_MAPPINGS, "com.mua.cloud.testm.models.events
.TestEvent:com.mua.cloud.testm.models.events
.TestEvent");


   return config;
}

1. Consumers do not use serializers. Remove that config. 2) Do you really need type information? Python doesn't need to add it, and you dont really need extra configs if you use StringDeserializer for the value, then you can parse JSON manually — OneCricketeer, Jul 18 '23 at 23:09

score 0 · Accepted Answer · answered Aug 13 '23 at 02:22

Now, I've found a solution.

I've removed all extra configurations from Spring Boot, other than basic serializer and deserializer for kafka.

Now,

right after receiving the message, i'm sanitizing the header
loading JSON data
work with that data
return data with payload and headers (we have a __TypeId__ field, which is really important)

Relevant codes

receiving and sending

for message in consumer:
    send(
         payload=load_message(message.value),
         headers=sanitize_headers(message.headers)
    )

sending

def send(payload: Dict, headers: list[tuple[str, Any]]) -> None:
   // work with these data
   producer = KafkaProducer(bootstrap_servers=broker)
   producer.send(THUMBNAIL_REPLY_TOPIC, 
   value=encode_payload(payload_updated), headers=headers)

misc

def sanitize_headers(headers: list[tuple[Any, Any]]) -> list[tuple[str, Any]]:
   return [(str(x[0]), x[1]) for x in headers]

def encode_payload(req: Dict) -> Any:
   return json.dumps(req).encode("utf-8")

def load_message(value):
   return json.loads(value)

If your ingress and egress data are of different types, just inspect thoese headers, and do a quick string replace on __TypeId__

Voila!! Everything under control.

Python to Spring Boot using Kafka: No type information in headers and no default type provided

1 Answers1

Relevant codes