I have a Kafka
consumer written in Java
and SpringBoot
.
I am using WebFlux
in order to make a call to trigger some actions on a third party server (and waiting the result of course).
This server has rate limit that is limiting me from making a lot of requests in a short time.
In order to prevent failures I intend to keep on trying calling the server using WebFlux backoff
:
webClientBuilder.build()
.get()
...
.retryWhen(getRetryPolicyOnTooManyRequests())
...
private RetryBackoffSpec getRetryPolicyOnTooManyRequests() {
return Retry.backoff(20, Duration.ofSeconds(retryBackoffMinimumSeconds))
.filter(this::is429Error);
}
private boolean is429Error(Throwable throwable) {
return throwable instanceof WebClientResponseException
&& ((WebClientResponseException) throwable).getStatusCode() == HttpStatus.TOO_MANY_REQUESTS;
}
My questions are about the behavior I should expect from my kafka:
- What will happen when I'll be
backoff
ing one of my calls? Will I be blocking the thread? Will a new thread be opened to process another message? - If I got the default consumer configurations (
max.poll.records
=500,max.poll.interval.ms
=30000) and my backoff time will get to 5 minutes will the kafka group be rebalanced? - If so, is there a smarter way to tackle this issue so I won't get rebalanced
each time, other than just putting a super high number in
max.poll.interval.ms