How to make the application self-recover from Reactor Kafka's RetriableCommitFailedException caused by request timeout?

Question

I have a Kafka processor that is defined like this.

import org.apache.kafka.clients.consumer.ConsumerConfig
import org.apache.kafka.common.serialization.StringDeserializer
import org.slf4j.LoggerFactory
import org.springframework.context.annotation.Bean
import org.springframework.stereotype.Component
import reactor.core.publisher.Mono
import reactor.core.scheduler.Schedulers
import reactor.kafka.receiver.KafkaReceiver
import reactor.kafka.receiver.ReceiverOptions
import reactor.kafka.receiver.ReceiverRecord
import reactor.kotlin.core.publisher.toMono
import reactor.util.retry.Retry
import java.time.Duration
import java.util.*

@Component
class KafkaProcessor {
    private val logger = LoggerFactory.getLogger(javaClass)
    
    private val consumerProps = hashMapOf(
        ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG to StringDeserializer::javaClass,
        ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG to StringDeserializer::javaClass,
        ConsumerConfig.GROUP_ID_CONFIG to "groupId",
        ConsumerConfig.AUTO_OFFSET_RESET_CONFIG to "earliest",
        ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG to "localhost:9092"
    )

    private val receiverOptions = ReceiverOptions.create<String, String>(consumerProps)
        .subscription(Collections.singleton("some-topic"))
        .commitInterval(Duration.ofSeconds(1))
        .commitBatchSize(1000)
        .maxCommitAttempts(1)

    private val kafkaReceiver: KafkaReceiver<String, String> = KafkaReceiver.create(receiverOptions)

    @Bean
    fun processKafkaMessages(): Unit {
        kafkaReceiver.receive()
            .groupBy { m -> m.receiverOffset().topicPartition() }
            .flatMap { partitionFlux ->
                partitionFlux.publishOn(Schedulers.elastic())
                    .concatMap { receiverRecord ->
                        processRecord(receiverRecord)
                            .map { it.receiverOffset().acknowledge() }
                    }
            }
            .retryWhen(
                Retry.backoff(3, Duration.ofSeconds(1))
                    .maxBackoff(Duration.ofSeconds(3))
                    .doBeforeRetry { rs ->
                        logger.warn("Retrying: ${rs.totalRetries() + 1}/3 due to ${rs.failure()}")
                    }
                    .onRetryExhaustedThrow { _, u ->
                        logger.error("All ${u.totalRetries() + 1} attempts failed with the last exception: ${u.failure()}")
                        u.failure()
                    }
            )
            .subscribe()
    }
    
    private fun processRecord(record: ReceiverRecord<String, String>): Mono<ReceiverRecord<String, String>> {
        return record.toMono()
    }
}

Sometimes, I got this error.

org.apache.kafka.clients.consumer.RetriableCommitFailedException: Offset commit failed with a retriable exception. You should retry committing the latest consumed offsets.
Caused by: org.apache.kafka.common.errors.TimeoutException: The request timed out.

The first retry looks like this.

Retrying: 1/3 due to org.apache.kafka.clients.consumer.RetriableCommitFailedException: Offset commit failed with a retriable exception. You should retry committing the latest consumed offsets

The second and third look like this.

Retrying: 2/3 due to reactor.core.Exceptions$ReactorRejectedExecutionException: Scheduler unavailable

Retrying: 3/3 due to reactor.core.Exceptions$ReactorRejectedExecutionException: Scheduler unavailable

And once all the 3 retries are exhausted, the message looks like this.

All 4 attempts failed with the last exception: reactor.core.Exceptions$ReactorRejectedExecutionException: Scheduler unavailable

When I do get that error, I need to restart the application in order to reconnect to the Kafka broker and commit the record.

I am aware that by setting maxCommitAttempts to 1 means that once it hits a RetriableCommitFailedException, it won't retry again. I thought that the retryWhen clause I put in the end of the processKafkaMessages() function would do the trick so that the pipeline can recover by itself.

The reason I set the maxCommitAttempts is because it does not have the retry with backoff as discussed here and the default 100 max commit attempts is done within 10ms. So, I thought I should write my own retry logic with a backoff.

The question is, how should I do the retry with backoff for the auto commit correctly? And is it possible to write a unit test for that using EmbeddedKafka?

Language: Kotlin

Reactor Kafka library: io.projectreactor.kafka:reactor-kafka:1.2.2.RELEASE

score 0 · Answer 1 · answered Mar 16 '22 at 14:18

retryWhen() merely attempts to re-subscribe. Since the Kafka consumer is in error state, it will reject the re-subscription. You need to defer the kafkaReceiver.receive() call, thus:

Flux.defer(() -> kafkaReceiver.receive())
            .groupBy { m -> m.receiverOffset().topicPartition() }
// etc

Thus re-subscription will call kafkaReceiver.receive() again and create a new consumer.

How to make the application self-recover from Reactor Kafka's RetriableCommitFailedException caused by request timeout?

1 Answers1