How can I wait until multiple WebClient Flux requests are finished?

Question

I want to:

subscribe to multiple endpoints returning Flux and output the messages I receive.
wait until all messages have been output from all endpoints before continuing.
avoid processing messages from multiple endpoints "together" (e.g. Flux.zip) because the endpoints will return an uneven number of messages and are not logically connected to each other
block forever if one or more endpoints generate an infinite number of messages

The following code satisfies 1 and 3 but not 2 and 4:

        Stream.of("http://service1.com", "http://service2.com", "http://service3.com")
                .forEach(service -> {
                    webClient.get()
                            .retrieve()
                            .bodyToFlux(String.class)
                            .map(message -> service + ": " + message)
                            .subscribe(System.out::println);
                });

        System.out.println("Received all messages");

The line "Received all messages" should not be printed until all endpoints have finished. However, because subscribe is asynchronous, that line is instead printed almost immediately and my application continues instead of waiting.

What should I do differently?

score 3 · Answer 1 · answered Dec 23 '20 at 00:57

3

I believe the following code snippet achieves 3 out of 4 points in your question though I do not feel like I fully understand the 3rd requirement. Let me if this example meets what is needed and if not, what is missing.

        Flux.just("http://service1.com", "http://service2.com", "http://service3.com")
            .flatMap(url -> webClient.get()
                    .uri(url)
                    .retrieve()
                    .bodyToFlux(String.class)
                    .map(body -> url + ":" + body)
            )
            .collectList()
            .doOnNext(list -> LOG.info("Received all messages"))
            .subscribe(list -> LOG.info("" + list));

answered Dec 23 '20 at 00:57

Michael McFadyen

2,675
11
25

collectList is not a good idea if there is a chance for an infinite stream, for that log message a doOnComplete might be more idiomatic – Martin Tarjányi Dec 23 '20 at 08:39
also the question writer will probably need to use block or blockLast if this code is just simply run in a main method – Martin Tarjányi Dec 23 '20 at 08:43

score 0 · Answer 2 · answered Dec 26 '20 at 11:33

flatMap is one way to merge fluxes together but you can also use Flux.merge

    List<Flux<String>> individualResults =
        Stream.of("http://service1.com", "http://service2.com", "http://service3.com")
            .map(
                service ->
                    webClient //
                        .get()
                        .retrieve()
                        .bodyToFlux(String.class))
            .collect(toList());
    Flux<String> mergedResults = Flux.merge(individualResults);  // Will not complete until all individual Fluxes have completed.
    mergedResults //
        .doOnNext(System.out::println)
        .then()
        .block(); // block this thread until mergedResults completes
    System.out.println("Received all messages");

How can I wait until multiple WebClient Flux requests are finished?

2 Answers2