0

I'm new to RxJava, so would be great if someone could clarify this...

Consider the following flow:

  1. Generate/emit an Integer each time a value is requested by a subscriber.
  2. Group generated integers so that each value constitutes its own group. Note: the goal is to simulate a case, in which a very large number of groups is present in an input.
  3. Now for each group:
  • Collect values into a series of 1 second-long chunks and stop when the first empty chunk arrives. Note: in real world there will be more than one value per group, but here there will be exactly 2 buffers per group: 1 buffer with a single int and 1 empty buffer, which will effectively complete the sequence.
  • Merge all non-empty chunks (1 in this case) into a single list.
  • Print resulting list into a standard output and request 1 more element.

Primitive implementation of the flow above:

import static java.util.stream.Collectors.toList;
import io.reactivex.rxjava3.annotations.NonNull;
import io.reactivex.rxjava3.core.Flowable;
import io.reactivex.rxjava3.core.FlowableSubscriber;
import org.reactivestreams.Subscription;

import java.util.List;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.stream.Stream;

public static void main(String[] args) {
final AtomicInteger ai = new AtomicInteger();
Flowable<Integer> flowable =
        Flowable.generate(emitter -> emitter.onNext(ai.getAndIncrement()));
flowable.groupBy(value -> value, value -> value, false, 1)
        .flatMap(grouped -> grouped.buffer(1, TimeUnit.SECONDS)
                .takeWhile(list -> !list.isEmpty())
                .reduce((list1, list2
                ) -> Stream.concat(list1.stream(), list2.stream()).collect(toList()))
                .toFlowable(), false, Integer.MAX_VALUE, 1)
        .subscribe(
                new FlowableSubscriber<>() {
                    private Subscription s;

                    @Override
                    public void onSubscribe(@NonNull Subscription s) {
                        this.s = s;
                        s.request(1);
                    }

                    @Override
                    public void onNext(List<Integer> groupedValues) {
                        assert groupedValues.size() == 1;
                        System.out.println(groupedValues);
                        s.request(1);
                    }

                    @Override
                    public void onError(Throwable t) {
                        t.printStackTrace();
                    }

                    @Override
                    public void onComplete() {
                        System.out.println(
                                "Done.");
                    }
                }); }

As you can see, I create only 1 custom subscriber, which requests 1 item upon subscription creation and 1 more each time a list is processed. Though flatMap's maxConcurrency is set to Integer.MAX_VALUE (since I want to handle as many groups as possible) - all other "hints" (such as groupBy's and flatMap's bufferSize) are set to 1 and no other buffering (such as onBackpressureBuffer, which "buffers a limited number of items from the current Flowable and allows it to emit as fast it can") is requested.


So problem is that number of times of onNext's invocations (and, hence, requests for a value by it) is way much lower than a number of invocations of a lambda that I passed to Flowable.generate - by more than 40,000 most of the times, and I'd like it to be at least the same order of magnitude as the number of elements requested via Subscrtiption.request. As I can judge by the lambda's callstack, each time a group is created, it invokes io.reactivex.rxjava3.internal.operators.flowable.FlowableGroupBy.GroupBySubscriber#onSubscribe, which, in turn, invokes org.reactivestreams.Subscription#request passing bufferSize (configured as 1 in the code snippet above) to it, so the process is pretty much self-perpetuating, so to speak. I can't specify bufferSize as 0, neither can I restrict the number of the groups being processed at the same time: finite maxConcurrency value quickly leads to MissingBackpressureException...

Am I missing something fundamental here or there is no way to truly apply back pressure in such a use case?

Thanks in advance.

Alexander
  • 131
  • 4
  • Which exact RxJava version are you using? – akarnokd Jul 20 '20 at 06:25
  • @akarnokd io.reactivex.rxjava3:rxjava:3.0.4 – Alexander Jul 20 '20 at 12:43
  • @akarnokd, should I create a github issue for this, or it is an intended behavior? – Alexander Jul 24 '20 at 16:31
  • You have a `takeWhile` that cuts a group short and can result in skipped values if there is a time gap due to asynchronicity introduced by the timed buffer. Does your setup hang (no error and no completion)? If not, you may need to redesign the flow and avoid canceling groups. – akarnokd Jul 24 '20 at 20:49
  • Yes, and I'm completely fine with considering a group completed once there're no emissions for it for 1 second (that's actually the goal of a real-world task). It does not hang for more than 1 sec and works fine (except for that huge difference in how many elements downstream requested vs. how many times upstream is asked for a new value). I would be OK with "buffer until timeout" feature (please see https://github.com/ReactiveX/RxJava/issues/6417) - but it is not implemented in RxJava3... – Alexander Jul 24 '20 at 22:06

0 Answers0