How to strip objects inside an array in Spring Webflux in order to get each object through an event

Question

What I want to achieve is to stream the objects under the JSON property "ranges" through flux events in order to apply a backpreasure mechanism.

I have a Json string which looks like this:

{
    "name": "the name",
    "id": "34343",
    "ranges": [
        {
            "range": "1 to 1000",
            "value": "11",
        },
        {
            "range": "1000-2000",
            "value": "12",
        },

I also have the java mapping in a class:

public class WholeJson {

    @JsonProperty("name")
    String name;

    @JsonProperty("id")
    String id;

    @JsonProperty("ranges")
    List<Range> ranges;
...
}

public class Range {
    private final String range;
    private final String value;

    public IpRange(@JsonProperty(value = "range") String range,
                   @JsonProperty(value = "value") String value) {

        this.range= range;
        this.value= value;
    }
...
...
}

How can I "strip off" the array and get only the list of Json Objects through the webclient, so that I can process the elements inside the array through more than just one event?

I used the webclient to return a list of Range object, which looked plausible to me.

...
...
    Mono<List<Range>> response = client.get()
                   .retrieve()
                   .bodyToMono(WholeJson.class)
                   .map(WholeJson::getRanges);

     Flux<List<Range>> secondresponse = Flux.from(response);

     secondresponse.subscribe(
                data -> onNext(data), // onNext
                err -> onError(err),  // onError
                () -> onComplete() // onComplete
        );
    }

    private static <T> void onNext(T data) {
        System.out.println("onNext: Data received: " + data);
    }

    private static <T> void onError(Throwable err) {
        System.out.println("onError: Exception occurred: " + err.getMessage());
    }

    private static <T> void onComplete() {
        System.out.println("onComplete: Completed!");
    }

But the output in the console shows that only one element is returned, that is the array (so the array is in fact a mono representing 1 element)

onNext: Data received: [range: 1-1000
value: 11
range: 1000-2000
value: 12]

What I expected was, that I'll receive an onNext-Event for each of the objects. What can I do to get the objects inside the array processed by events?

Thank you very much in advance for your help.

score 0 · Accepted Answer · answered Aug 29 '23 at 01:17

0

You create a flux from a single object which happen to be a List instance but Flux doesn't know that you want to process the elements of it.

You have to convert to a Flux of its items like that:

Flux<Range> secondresponse = Flux.from(response)
    .flatMap(Flux::fromIterable);

Or in other way:

Mono<WholeJson> response = client.get()
               .retrieve()
               .bodyToMono(WholeJson.class);

Flux<Range> secondresponse = Flux.from(response)
    .flatMap(json -> Flux.fromIterable(json.getRanges));

answered Aug 29 '23 at 01:17

Igor Artamonov

35,450
10
82
113

Thank you Igor, your answer answered my question. – Horseman Aug 29 '23 at 06:25
Since I have you here, Igor, I would like to ask you another question. Do you think, this flux mechanism combined with a backpreasure method can reduce the memory footprint of filtering a big json (100 MB or more of size)? – Horseman Aug 29 '23 at 06:45
When you parse such a JSON you already put it into the memory. Flux can process it in a streaming way and save the memory on these steps but it may be too late, and you have to find a way to to parse JSON as a stream before it passing it to Reactor. It's possible, even manually splitting the `InputStream` with JSON by `},{` as a delimiter, or using other libs that can do that. – Igor Artamonov Aug 29 '23 at 13:41
I used RestTemplate and Jackson parser following this example: https://dev.to/ratulsharker/consume-large-json-response-in-spring-421j Is this the way you were thinking of or is there a way to use this way of json parsing with the webclient? – Horseman Aug 31 '23 at 20:48
Yes, looks like a way to do that. It's a bit complex code to parse token by token, but it allows to do it in a streaming way and process each item / chunk with Flux – Igor Artamonov Aug 31 '23 at 21:51
I combined the parsing from the first url with the best practise from https://www.baeldung.com/spring-reactive-read-flux-into-inputstream and got similar results as with the RestTemplate. The Memory usage with the WebClient is just 2 MB above the memory usage with the RestTemplate. When parsing a 100 MB file, I have to set the heap for the RestTemplate app to 16 MB and for the WebClient app to 18 MB. Thank you Igor for this inspiring conversation. – Horseman Sep 01 '23 at 23:15

How to strip objects inside an array in Spring Webflux in order to get each object through an event

1 Answers1