RxJava: How to implement a slow consumer with buffer and a fast consumer to the same source?

Question

While playing with various backpressure scenarios, I implemented a case where one subscriber is slow with a buffer, while another one consumes whatever is thrown to it. That was using Scala and Akka Streams. You can see the code here if you want to and the test that runs it here.

I usually try to develop a RxJava version for comparison but I got stuck on this one. In Akka Streams, I can build a graph with one source that broadcasts on 2 channels, and have a slow sink and a fast sink consume from those channels. Each channel can independently apply buffering and throttling. In RxJava, there is the share operator for broadcasting but the buffering and throttling logic is not on the Subscriber, but on the Observable. Thus I'm not sure how to apply buffering and throttling and not have both subscribers affected. Both Akka Streams and RxJava being implementation of Rx, I'm hoping there's a way to get what I want.

Here's a pictorial version of what I'm trying to do.

Do you have some RxJava code attempting this? Even if it's incomplete hence your question? — tmn, Sep 01 '16 at 23:16
@ThomasN. Not yet because I can't figure out how to achieve this. Writing code is the easy part. — Abhijit Sarkar, Sep 02 '16 at 00:58

tmn · Answer 1 · 2016-09-02T02:48:08.760

Something like this?

import rx.Observable;
import rx.observables.ConnectableObservable;

import java.util.concurrent.TimeUnit;

public class Test {
    public static void main(String[] args) {

        //emits Long every 100 milliseconds, and publishes to all Subscribers simultaneously through ConnectableObservable
        ConnectableObservable<Long> source = Observable.interval(100, TimeUnit.MILLISECONDS).publish();

        //buffers emissions into Lists within 1 second durations, and first Subscriber prints them
        source.buffer(1,TimeUnit.SECONDS).subscribe(System.out::println);

        //no buffering, just push emissions directly to this second Subscriber which prints them
        source.subscribe(System.out::println);

        //start firing emissions now both Subscribers are connected
        source.connect();

        //sleep to keep program alive for 10 seconds
        try {
            Thread.sleep(10000);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}

The Subscriber has no notion of throttling or any operators. Those are done on the Observable side through various operators which yield different Observable implementations. The Subscriber is pretty dumb and simply consumes the emission as the final step in an Observable chain. It is agnostic to which thread the emission comes on, much less whether its been throttled or not by an Observable passing items to it.

You might want to read this to see all the different throttling, windowing, etc. operators https://github.com/ReactiveX/RxJava/wiki/Backpressure — tmn, Sep 02 '16 at 02:40
Regarding throttling, `debounce(1, SECONDS)` doesn't slow the publisher down, it just drops the items. I didn't find anything in the docs to safely and easily apply backpressure. The only option is to implement my own. — Abhijit Sarkar, Sep 02 '16 at 05:51
I'm not a backpressure expert because I don't use it often. But my understanding is some source Observables cannot be slowed down just by their very nature. For instance, this time interval cannot be slowed down. Neither can UI events. — tmn, Sep 02 '16 at 16:37
See the gist [here](https://gist.github.com/abhijitsarkar/5306c82fcf62f5b95ad723436d2ed5d1) based on the blog [How a subscriber establishes “reactive pull” backpressure](https://github.com/ReactiveX/RxJava/wiki/Backpressure#how-a-subscriber-establishes-reactive-pull-backpressure). Doesn't work, pub doesn't slow down. I switched to RxJava 2.0 for better backpressure support. Didn't help. — Abhijit Sarkar, Sep 04 '16 at 00:20

score 0 · Answer 2 · answered Sep 26 '16 at 02:36

You should be able decorate the shared() observable in different ways for different subscribers if you want different backpressure behaviors.

For instance

Observable<Integer> source = Observable.interval(0, 1, TimeUnit.SECONDS).share();

// Naked source for fast consumers.
Observable<Integer> fast = source; 

// Buffer for slow consumers that use backpressure.
Observable<Integer> slow = source.onBackpressureBuffer();

Subscribers to fast and slow above will ultimately use the same shared source.

Note that fast does not respond to backpressure because interval does not respond to backpressure.

There are different flavors of onBackpressureXXX() that you can get different behaviors with.

I'm reasonably sure that I looked at `onBackpressureBuffer` and it didn't give me the same behavior I was able to get using Akka Streams. I've gotten busy with other things since posting the question but I've it on my radar to check and confirm whether your suggestion works. — Abhijit Sarkar, Oct 03 '16 at 01:58

RxJava: How to implement a slow consumer with buffer and a fast consumer to the same source?

2 Answers2