Error Handling - protobuf message to DLT instead of Retry in case of connection exception

Question

I've implemented an KafkaListener using SpringBoot 2.7.6, Confluent platform and now I need to implement the Error Handler for it.

The listener manages to pick up a protobuf topic message and POST the payload to an HTTP endpoint properly. But in case of the java.net.ConnectException happens I need to send the same protobuf message to a DLT instead of Retry.

I implemented this using the following Listener:

@Component
class ConsumerListener(
    private val apiPathsConfig: ApiPathsConfig,
    private val myHttpClient: MyHttpClient,
    @Value("\${ingestion.config.httpClientTimeOutInSeconds}") private val httpRequestTimeout: Long
) {
    val log: Logger = LoggerFactory.getLogger(ConsumerListener::class.java)

    @RetryableTopic(
        attempts = "4",
        backoff = Backoff(delay = 5000, multiplier = 2.0),    //TODO: env var?
        autoCreateTopics = "false",
        topicSuffixingStrategy = TopicSuffixingStrategy.SUFFIX_WITH_INDEX_VALUE,
        timeout = "3000",    //TODO: env var?
        dltStrategy = DltStrategy.FAIL_ON_ERROR
    )
    @KafkaListener(
        id = "ingestionConsumerListener",
        topics = ["#{'\${ingestion.config.topic.name}'}"],
        groupId = "#{'\${ingestion.consumer.group.id}'}",
        concurrency = "#{'\${ingestion.config.consumer.concurrency}'}"
    )
    fun consume(ingestionHttpRequest: ConsumerRecord<String, HttpRequest.HttpRequest>) {

        ...
        try {
            val response: HttpResponse<Void> = myHttpClient.send(request, HttpResponse.BodyHandlers.discarding())
            if (response.statusCode() in 400..520) {
                val ingestionResponseError = "Ingestion response status code [${response.statusCode()}] - headers [${response.headers()}] - body [${response.body()}]"
                log.error(ingestionResponseError)
                throw RuntimeException(ingestionResponseError)
            }
        } catch (e: IOException) {
            log.error("IOException stackTrace : ${e.printStackTrace()}")
            throw RuntimeException(e.stackTrace.contentToString())
        } catch (e: InterruptedException) {
            log.error("InterruptedException stackTrace : ${e.printStackTrace()}")
            throw RuntimeException(e.stackTrace.contentToString())
        } catch (e: IllegalArgumentException) {
            log.error("IllegalArgumentException stackTrace : ${e.printStackTrace()}")
            throw RuntimeException(e.stackTrace.contentToString())
        }

    }
    ...
}

When the java.net.ConnectException happens the DeadLetterPublishingRecoverFactory show this:

15:19:44.546 [kafka-producer-network-thread | producer-1] INFO  org.apache.kafka.clients.producer.internals.TransactionManager - [Producer clientId=producer-1] ProducerId set to 3330155 with epoch 0
15:19:44.547 [ingestionConsumerListener-2-C-1] ERROR org.springframework.kafka.retrytopic.DeadLetterPublishingRecovererFactory$1 - Dead-letter publication to ingestion-topic-retry-0failed for: ingestion-topic-5@32
org.apache.kafka.common.errors.SerializationException: Can't convert value of class com.xxx.ingestion.IngestionHttpRequest$HttpRequest to class org.apache.kafka.common.serialization.StringSerializer specified in value.serializer

...
Caused by: java.lang.ClassCastException: class com.xxx.ingestion.IngestionHttpRequest$HttpRequest cannot be cast to class java.lang.String (com.xxx.ingestion.IngestionHttpRequest$HttpRequest is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')
    at org.apache.kafka.common.serialization.StringSerializer.serialize(StringSerializer.java:29)

Please, how to resend the protobuf message to a DLT instead of Retry in case of ConnectionException and how to keep the Retry in case of the the HTTP endpoint response 4xx or 5xx code?

Please user:2756547

Answer 1

You either need a protobuf serializer to re-serialize the data, or don't deserialize the prototbuf in a deserializer, use a ByteArrayDeserializer and convert it in your listener instead; then use a ByteArraySerializer.

You can also configure certain exception types to go straight to the DLT.

EDIT

See https://docs.spring.io/spring-kafka/docs/current/reference/html/#retry-topic-global-settings

@Configuration
public class MyRetryTopicConfiguration extends RetryTopicConfigurationSupport {

    @Override
    protected void manageNonBlockingFatalExceptions(List<Class<? extends Throwable>> nonBlockingFatalExceptions) {
        nonBlockingFatalExceptions.add(MyNonBlockingException.class);
    }

}

/**
 * Override this method to manage non-blocking retries fatal exceptions.
 * Records which processing throws an exception present in this list will be
 * forwarded directly to the DLT, if one is configured, or stop being processed
 * otherwise.
 * @param nonBlockingRetriesExceptions a {@link List} of fatal exceptions
 * containing the framework defaults.
 */
protected void manageNonBlockingFatalExceptions(List<Class<? extends Throwable>> nonBlockingRetriesExceptions) {
}