I'm new to RxJava, so would be great if someone could clarify this...
Consider the following flow:
- Generate/emit an Integer each time a value is requested by a subscriber.
- Group generated integers so that each value constitutes its own group. Note: the goal is to simulate a case, in which a very large number of groups is present in an input.
- Now for each group:
- Collect values into a series of 1 second-long chunks and stop when the first empty chunk arrives. Note: in real world there will be more than one value per group, but here there will be exactly 2 buffers per group: 1 buffer with a single int and 1 empty buffer, which will effectively complete the sequence.
- Merge all non-empty chunks (1 in this case) into a single list.
- Print resulting list into a standard output and request 1 more element.
Primitive implementation of the flow above:
import static java.util.stream.Collectors.toList;
import io.reactivex.rxjava3.annotations.NonNull;
import io.reactivex.rxjava3.core.Flowable;
import io.reactivex.rxjava3.core.FlowableSubscriber;
import org.reactivestreams.Subscription;
import java.util.List;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.stream.Stream;
public static void main(String[] args) {
final AtomicInteger ai = new AtomicInteger();
Flowable<Integer> flowable =
Flowable.generate(emitter -> emitter.onNext(ai.getAndIncrement()));
flowable.groupBy(value -> value, value -> value, false, 1)
.flatMap(grouped -> grouped.buffer(1, TimeUnit.SECONDS)
.takeWhile(list -> !list.isEmpty())
.reduce((list1, list2
) -> Stream.concat(list1.stream(), list2.stream()).collect(toList()))
.toFlowable(), false, Integer.MAX_VALUE, 1)
.subscribe(
new FlowableSubscriber<>() {
private Subscription s;
@Override
public void onSubscribe(@NonNull Subscription s) {
this.s = s;
s.request(1);
}
@Override
public void onNext(List<Integer> groupedValues) {
assert groupedValues.size() == 1;
System.out.println(groupedValues);
s.request(1);
}
@Override
public void onError(Throwable t) {
t.printStackTrace();
}
@Override
public void onComplete() {
System.out.println(
"Done.");
}
}); }
As you can see, I create only 1 custom subscriber, which requests 1 item upon subscription creation and 1 more each time a list is processed. Though flatMap
's maxConcurrency
is set to Integer.MAX_VALUE
(since I want to handle as many groups as possible) - all other "hints" (such as groupBy's and flatMap's bufferSize
) are set to 1 and no other buffering (such as onBackpressureBuffer
, which "buffers a limited number of items from the current Flowable
and allows it to emit as fast it can") is requested.
So problem is that number of times of onNext
's invocations (and, hence, requests for a value by it) is way much lower than a number of invocations of a lambda that I passed to Flowable.generate
- by more than 40,000 most of the times, and I'd like it to be at least the same order of magnitude as the number of elements requested via Subscrtiption.request
.
As I can judge by the lambda's callstack, each time a group is created, it invokes io.reactivex.rxjava3.internal.operators.flowable.FlowableGroupBy.GroupBySubscriber#onSubscribe
, which, in turn, invokes org.reactivestreams.Subscription#request
passing bufferSize
(configured as 1 in the code snippet above) to it, so the process is pretty much self-perpetuating, so to speak. I can't specify bufferSize
as 0, neither can I restrict the number of the groups being processed at the same time: finite maxConcurrency
value quickly leads to MissingBackpressureException
...
Am I missing something fundamental here or there is no way to truly apply back pressure in such a use case?
Thanks in advance.