Parallel execution of computationally expensive map

Question

I am new to the ReactiveX library (I use its scala variant, RxScala).

I have an Observable that emits values at high rate. I would like to apply a function to all values of the Observable (map). The function, which I use in map, is computationally rather expensive.

Is there a way to have a thread pool for computing the map phase in parallel?

Show some sample code to reproduce your problem. – Reut Sharabani Nov 20 '15 at 01:20 — Reut Sharabani, Nov 20 '15 at 01:20

score 7 · Accepted Answer · answered Nov 20 '15 at 00:12

Yep, there is a way to do this.

I would buffer the stream into chunks and distribute the load across cpus using Schedulers.computation() (which uses an Executor based on a thread pool with size equal to the number of available processors):

int chunkSize = 1000;
source
  .buffer(chunkSize)
  .flatMap(
    list -> 
      Observable
        .from(list)
        .map(expensive)
        .subscribeOn(Schedulers.computation()))
 ...

If the map operation is sufficiently expensive you might be just as performant without buffer:

source
  .flatMap(
    x -> 
      Observable
        .just(x)
        .map(expensive)
        .subscribeOn(Schedulers.computation()))

Parallel execution of computationally expensive map

1 Answers1