Reactive Streams: How to wait for all publishers, by key?

Question

Suppose I have 3 publishers and 1 processor. The publishers emits items in the form {key: <integer>, value: <object>, publisher_id: <string>}.

The publishers makes IO operations, so:

On the one hand, I'd like the publishers to work on (roughly) N items at a given moment.
On the other hand, I'd like the consumer to merge the items to one single record (i.e. {key: <integer>, values: <list>})

I've actually already implemented a FluxProcessor that has an internal storage (ConcurrentHashMap) to keep all the items. It manually request() new items whenever CAPACITY wasn't reached.

I'd like to know if there's a built-in functionality to do that with RxJava(2)/ Spring Reactor API?

score 1 · Accepted Answer · answered Jun 27 '19 at 08:19

1

Use merge, rebatchRequests and toMultimap with RxJava 2:

Flowable<KeyValuePublisher> source1 = ...
Flowable<KeyValuePublisher> source2 = ...
Flowable<KeyValuePublisher> source3 = ...

Flowable.merge(
    source1.rebatchRequests(N),
    source2.rebatchRequests(N),
    source3.rebatchRequests(N)
)
.toMultimap(kvp -> kvp.key, kvp -> kvp.value)
subscribe(map -> System.out.println(map));

answered Jun 27 '19 at 08:19

akarnokd

69,132
14
157
192

Thank you akarnokd! – IsaacLevon Jun 28 '19 at 19:47
I have to admit that this not actually what I wanted, as `toMultiMap` is a "sink" which ends the `Flowable` – IsaacLevon Jun 30 '19 at 11:37
That do you mean by "ends the Flowable". You can [convert between types](https://github.com/ReactiveX/RxJava#converting-to-the-desired-type). – akarnokd Jul 01 '19 at 09:20
Do you happen to know what's the equivalent in Spring reactor? – IsaacLevon Jul 08 '19 at 11:57

Reactive Streams: How to wait for all publishers, by key?

1 Answers1