How to call several http requests in parallel(with retry functionality) and await when all of them are finish?

Question

For now my code looks like this:

   List<Mono<ResponseEntity<String>>> response = queue.stream()
            .map(req-> webClient
                    .post()
                    .bodyValue(req)
                    .retrieve()
                    .toEntity(String.class)
            )
            .collect(Collectors.toList());

How could I await the moment when all responses will be accepted ?

If some of requests are failed I would like to retry only them.

How could I achieve it?

I guess you want automatic retries? Or you want to retry them explicitly? — Haris Osmanagić, Nov 18 '19 at 11:14
@Haris Osmanagić I prefer automatic retry. WebClient supports it out of the box for single request — gstackoverflow, Nov 18 '19 at 11:16
You can definitely use an Executor, and submit all of your requests as tasks. If you can retry a single request easily using WebClient, then waiting for all them to finish is super easy. — Haris Osmanagić, Nov 18 '19 at 11:22

score 1 · Answer 1 · answered Nov 18 '19 at 12:09

1

The simplest and strait-forward solution would be to write a code that sends a single request with retry and returns only after it completed successfully or ran out of the max limit of reties. After that Wrap that code as an Implementation of Runnable and use ExecutorService to submit all of them. Collect the Futures into a collection and check when all of them completed.

answered Nov 18 '19 at 12:09

Michael Gantman

7,315
2
19
36

1

I don't think it makes sense to use ExecutorService and Future when you have Project Reactor at your disposal. – Martin Tarjányi Nov 18 '19 at 18:56

score 1 · Accepted Answer · answered Nov 18 '19 at 18:54

Rather than going with ExecutorService suggested by another answer, I'd recommend using the capabilities of Mono and Flux which provides a more idiomatic solution:

Mono<List<String>> response = Flux.fromIterable(queue)
                                  .flatMap(this::callHttp)
                                  .collectList();

private Mono<String> callHttp(String req)
{
    return webClient
            .post()
            .syncBody(req)
            .retrieve()
            .bodyToMono(String.class)
            .retry(3); // retries failed requests at most 3 times
}

How to call several http requests in parallel(with retry functionality) and await when all of them are finish?

2 Answers2