2

I have a Kafka consumer written in Java and SpringBoot. I am using WebFlux in order to make a call to trigger some actions on a third party server (and waiting the result of course). This server has rate limit that is limiting me from making a lot of requests in a short time. In order to prevent failures I intend to keep on trying calling the server using WebFlux backoff:

webClientBuilder.build()
.get()
...
.retryWhen(getRetryPolicyOnTooManyRequests())
...

private RetryBackoffSpec getRetryPolicyOnTooManyRequests() {
    return Retry.backoff(20, Duration.ofSeconds(retryBackoffMinimumSeconds))
        .filter(this::is429Error);
}

private boolean is429Error(Throwable throwable) {
    return throwable instanceof WebClientResponseException
        && ((WebClientResponseException) throwable).getStatusCode() == HttpStatus.TOO_MANY_REQUESTS;
}

My questions are about the behavior I should expect from my kafka:

  1. What will happen when I'll be backoffing one of my calls? Will I be blocking the thread? Will a new thread be opened to process another message?
  2. If I got the default consumer configurations (max.poll.records=500, max.poll.interval.ms=30000) and my backoff time will get to 5 minutes will the kafka group be rebalanced?
  3. If so, is there a smarter way to tackle this issue so I won't get rebalanced each time, other than just putting a super high number in max.poll.interval.ms
Tom Carmi
  • 385
  • 1
  • 5
  • 18

0 Answers0