How to retry failed ConsumerRecord in reactor-kafka

Question

I am trying on reactor-kafka for consuming messages. Everything else work fine, but I want to add a retry(2) for failing messages. spring-kafka already retries failed record 3 times by default, I want to achieve the same using reactor-kafka.

I am using spring-kafka as a wrapper for reactive-kafka. Below is my consumer template:

reactiveKafkaConsumerTemplate
                .receiveAutoAck()
                .map(ConsumerRecord::value)
                .flatMap(this::consumeWithRetry)
                .onErrorContinue((error, value)->log.error("something bad happened while consuming : {}", error.getMessage()))
                .retryWhen(Retry.backoff(30, Duration.of(10, ChronoUnit.SECONDS)))
                .subscribe();

Let us consider the consume method is as follows

public Mono<Void> consume(MessageRecord message){
   return Mono.error(new RuntimeException("test retry"); //sample error scenario
}

I am using the following logic to retry the consume method on failure.

public Mono<Void> consumeWithRetry(MessageRecord message){
   return consume(message)
          .retry(2);
}

I want to retry consuming the message if the current consumer record fails with exception. I have tried to wrap the consume method with another retry(3) but that does not serve the purpose. The last retryWhen is only for retrying subscription on kafka rebalances.

@simon-baslé @gary-russell

score 2 · Accepted Answer · answered Oct 01 '21 at 09:40

Previously while retrying I was using the below approach:

public Mono<Void> consumeWithRetry(MessageRecord message){
   return consume(message)
          .retry(2);
}

But it was not retrying. After adding Mono.defer, the above code works and adds required retry.

public Mono<Void> consumeWithRetry(MessageRecord message){
   return Mono.defer(()->consume(message))
          .retry(2);
}

How to retry failed ConsumerRecord in reactor-kafka

1 Answers1

Linked