I am using spring-kafka 2.8.9 and kafka-clients 2.8.1 . I want to skip a message which is failed to de-serialize . Since setErrorHandler is deprecated , I tried using CommonErrorHandler . But I am not sure how to skip current error message and move to next record . The only option I can see is using pattern matching by extracting relevant details from below line like offset and partition .
org.apache.kafka.common.errors.SerializationException: Error deserializing key/value for partition test-0 at offset 1. If needed, please seek past the record
Is there any other way like RecordDeserializationException to get necessary information from the exception or any other means without pattern matching . I can not upgrade to kafka 3.X.X .
My config
@Bean
public ConsumerFactory<String, Farewell> farewellConsumerFactory()
{
groupId = LocalTime.now().toString();
Map<String, Object> props = new HashMap<>();
props.put( ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapAddress);
props.put( ConsumerConfig.GROUP_ID_CONFIG, groupId);
props.put(JsonDeserializer.TRUSTED_PACKAGES, "*");
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"earliest");
return new DefaultKafkaConsumerFactory<>(props,new StringDeserializer(),new JsonDeserializer<>(Farewell.class));
}
@Bean
public ConcurrentKafkaListenerContainerFactory<String, Farewell> farewellKafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, Farewell> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setCommonErrorHandler(new CommonErrorHandler()
{
@Override
public void handleOtherException(Exception thrownException, Consumer<?, ?> consumer, MessageListenerContainer container, boolean batchListener)
{
CommonErrorHandler.super.handleOtherException(thrownException, consumer, container, batchListener);
}
});
factory.setConsumerFactory(farewellConsumerFactory());
return factory;
}
My listener class
@KafkaListener(topics = "${topicId}",
containerFactory = "farewellKafkaListenerContainerFactory")
public void farewellListener(Farewell message) {
System.out.println("Received Message in group " + groupId + "| " + message);
}
Domain class
public class Farewell {
private String message;
private Integer remainingMinutes;
public Farewell(String message, Integer remainingMinutes)
{
this.message = message;
this.remainingMinutes = remainingMinutes;
}
// standard getters, setters and constructor
}
I have checked these links How to skip a msg that have error in kafka when i use ConcurrentMessageListenerContainer?
