3

Is it possible to rate limit webclient requests for a url or any requests after a defined number of retries using resilience4j or otherwise?

Following is a sample code.

webClient.post()
         .bodyValue(...)
         .header(...)
         .uri(...)
         .retrieve()
         .bodyToMono(Void.class)
         .retryWhen(...) // every 15 mins, for 1 day
         .subscribe();

Example use is say 10000 requests in a day need to be sent to 100 different urls. There are retries for case when a url is temporary unavailable.

But if a url comes back up after few hours, it would get accumulated large number of requests which I would like to rate limit.

Effectively I don't want to rate limit the whole operation but for specific urls which are not available for a long time or rate limit requests which have been retried 'x' number of times. Is there a way to achieve this?

Not to be confused with circuit breaker requirement, it's not exactly an issue to keep retrying every few mins atleast in the context.

flamethrower
  • 31
  • 2
  • 1
    for url which are not available you can use circuitbreaker – silentsudo Aug 25 '21 at 07:16
  • yes, ideally instead of retries it should be using circuitbreaker to check whether a url is up and than proceed with more requests. but once it's back you would want to ratelimit still. also ideally don't want a universal circuitbreaker, but one for each url sorta does that make sense – flamethrower Aug 25 '21 at 07:41
  • what's the matter with resilience4j RateLimiter? did you have some issues with it? – Martin Tarjányi Aug 26 '21 at 06:40
  • @MartinTarjányi I'm not sure how to apply ratelimiter for my use case. One of the option I considered was to somehow rate limit each url seperately (not rate limiting the full function but rate limit each one individually), which implies having as rate limiter instances as the urls, probably doens't make sense. Can't figure out how to apply the transformDeffered with eg. retryWhen(Retry.fixedDelay(a,b).doAfterRetry(e -> {if (e.totalRetries>1000). ??) Also I had tried R4J's (@Retry @RateLimiter) and retryWhen().transformDeffered(RateLimiter), latter somehow hadn't worked – flamethrower Aug 26 '21 at 08:54
  • As you wrote, you can create RateLimiter per url (or per host, or per path). It makes sense if that's your requirement. I'm not sure I completely get the second part of your question but I'd consider putting the RateLimiter before the retry, so: `transformDeferred(RateLimiter).retryWhen()`. Is it a requirement to apply rate limiter only when retries happen? Is it a problem if it is applied for normal/successful requests? That adds lots of complexity and I'm not sure about the benefit. – Martin Tarjányi Aug 26 '21 at 09:07

0 Answers0