How can I avoid using a schema registry for my Dead Letter Topics?

Question

I have a KafkaListener consuming Avro messages from Kafka. If the logic that processes the consumed message throws an exception the error handler will send the message to a DLT. I want to send that message as a JSON and ignore any schema registry stuff. I'm running into problems because if I try to use a JSON serializer it fails to serialize since it's an Avro object.

error handler

@Bean
  public DefaultErrorHandler errorHandler(final KafkaTemplate<String, ?> template) {

    DefaultErrorHandler handler =
        new DefaultErrorHandler(
            new DeadLetterPublishingRecoverer(template), new FixedBackOff(0L, 0L));
    handler.addNotRetryableExceptions(
        IllegalArgumentException.class, SaveFactFailedException.class);
    return handler;
  }

Producer Configs

producer:
    key-serializer: org.apache.kafka.common.serialization.StringSerializer
    value-serializer: com.amazonaws.services.schemaregistry.serializers.avro.AWSKafkaAvroSerializer

That's the first problem.

The second issue is the original listener and the DLT listener are both using the same Deserializer which is set in the application.yml. How can I configure different deserializers for each listener?

Consumer Configs

    consumer:
      enable-auto-commit: false
      group-id: mobile-layout-group
      key-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
      properties:
        spring.deserializer.key.delegate.class: org.apache.kafka.common.serialization.StringDeserializer
        spring.deserializer.value.delegate.class: com.amazonaws.services.schemaregistry.deserializers.avro.AWSKafkaAvroDeserializer
      auto-offset-reset: earliest

score 0 · Accepted Answer · answered Nov 28 '22 at 23:45

So I was able to figure it out. All I had to do was create a subclass of DeadLetterPublishingRecoverer and override the createProducer() function. Then I was able to make modifications to the record value. In my case I just made it null because I only needed the headers for my DLT. This article was helpful.

Spring Cloud Stream JSON Dead Letter Queue with Avro messages

How can I avoid using a schema registry for my Dead Letter Topics?

1 Answers1