Execute parallel tasks with RxJava and wait until the last one of them is completed

Question

Well, this is my first question in StackOverflow, but after several days fighting with RxJava I am unable to find other solution, I have tried many things by my own, digging along the documentation and other posts but I am not sure how exactly do what I need. I have tried with several combinations of flatMap, zip, merge and others, but always reaching to a dead end, and the closest achieved solution is the code below. I would appreciate any help or guidance.

I need a method that, given an input list, executes parallel calls with the different inputs of the list, and do not continue the execution until all the parallel calls have finished. It is also necessary to keep the results of the different executions for later uses (EDIT: In the same thread that started the execution).

public void parallelExecution(List<Integer> calls) {
  List<String> results = new ArrayList<>();
  logger.debug("Starting parallel executions");
  Observable.fromIterable(calls)
      .flatMap(val -> Observable.just(val).observeOn(Schedulers.io())
      .doOnNext(item -> results.add(simpleAsync(item).toString())))
      .subscribe(call -> logger.debug("Execution: {}", Thread.currentThread().getName()));
  logger.debug("Ending parallel executions");
  for (String x : results) {
    logger.debug("Results: {}", x);
  }
}

private Integer simpleAsync(Integer number) {
  Integer result = number * number;
  logger.info("Pre log {}: {}", Thread.currentThread().getName(), result);
  try {
    Thread.sleep(number * 500);
  } catch (Exception e) {
  }
  logger.info("Post log {}: {}", Thread.currentThread().getName(), result);
  return result;
}

The problem is that this code does not "waits" for the execution of the "simpleAsync" method, it finishes the execution with no "Results" logs (there are no results yet), and after that, the "Post log" traces appear executed in different threads:

Starting parallel executions
Ending parallel executions
Pre log RxCachedThreadScheduler-1: 1
Pre log RxCachedThreadScheduler-2: 4
Pre log RxCachedThreadScheduler-3: 9
Pre log RxCachedThreadScheduler-4: 16
Pre log RxCachedThreadScheduler-5: 25
Post log RxCachedThreadScheduler-1: 1
Execution: RxCachedThreadScheduler-1
Post log RxCachedThreadScheduler-2: 4
Execution: RxCachedThreadScheduler-2
Post log RxCachedThreadScheduler-3: 9
Execution: RxCachedThreadScheduler-3
Post log RxCachedThreadScheduler-4: 16
Execution: RxCachedThreadScheduler-4
Post log RxCachedThreadScheduler-5: 25
Execution: RxCachedThreadScheduler-5

If I delete the "observeOn" sentence, the method waits for the completion of the calls, but they are done sequentially (in the same thread):

Starting parallel executions
Pre log Default Executor-thread-9: 1
Post log Default Executor-thread-9: 1
Execution: Default Executor-thread-9
Pre log Default Executor-thread-9: 4
Post log Default Executor-thread-9: 4
Execution: Default Executor-thread-9
Pre log Default Executor-thread-9: 9
Post log Default Executor-thread-9: 9
Execution: Default Executor-thread-9
Pre log Default Executor-thread-9: 16
Post log Default Executor-thread-9: 16
Execution: Default Executor-thread-9
Pre log Default Executor-thread-9: 25
Post log Default Executor-thread-9: 25
Execution: Default Executor-thread-9
Ending parallel executions
Results: 1
Results: 4
Results: 9
Results: 16
Results: 25

Sanlok Lee · Accepted Answer · 2019-01-30T04:04:28.423

Have you tried using zip?

public void parallelExecution(List<Integer> calls) {

    logger.debug("Starting parallel executions");

    // Create an iterable observables
    List<Observable<Integer>> observables = calls.stream()
            .map(i -> {
                return Observable.fromCallable(() -> simpleAsync(i))
                        .subscribeOn(Schedulers.newThread());
            })
            .collect(Collectors.toList());


    Observable.zip(observables, objects -> { // Zip observables
                return Arrays.stream(objects)
                        .map(Object::toString)
                        .collect(Collectors.toList());
            })
            .doOnNext(results -> logger.debug("Ending parallel executions"))
            .subscribe(results -> { // Subscribe to the result.
                // Put your code that needs to "wait"
                for (String x : results) {
                    logger.debug("Results: {}", x);
                }
            });
}

The result will look something like this:

Starting parallel executions
Pre log RxNewThreadScheduler-3: 9
Pre log RxNewThreadScheduler-1: 1
Pre log RxNewThreadScheduler-2: 4
Pre log RxNewThreadScheduler-4: 16
Pre log RxNewThreadScheduler-5: 25
Post log RxNewThreadScheduler-1: 1
Post log RxNewThreadScheduler-2: 4
Post log RxNewThreadScheduler-3: 9
Post log RxNewThreadScheduler-4: 16
Post log RxNewThreadScheduler-5: 25
Ending parallel executions
Results: 1
Results: 4
Results: 9
Results: 16
Results: 25

EDIT: You can change the thread you want to listen to the result using observeOn. If you want to subscribe from the calling thread, for example, you can change the code to something like this (see these SO answers):

final BlockingQueue<Runnable> tasks = new LinkedBlockingQueue<>();
logger.debug("Starting parallel executions");

// Create an iterable observables
List<Observable<Integer>> observables = calls.stream()
        .map(i -> {
            return Observable.fromCallable(() -> simpleAsync(i))
                    .subscribeOn(Schedulers.newThread());
        })
        .collect(Collectors.toList());


Observable.zip(observables, objects -> { // Zip observables
            return Arrays.stream(objects)
                    .map(Object::toString)
                    .collect(Collectors.toList());
        })
        .doOnNext(results -> logger.debug("Ending parallel executions"))
        .observeOn(Schedulers.from(tasks::add)) // Add a scheduler with executor from the current thread
        .subscribe(results -> { // Subscribe to the result.
            // Put your code that needs to "wait"
            for (String x : results) {
                logger.debug("Results: {}", x);
            }
        });

try {
    tasks.take().run();
} catch (InterruptedException e) {
    e.printStackTrace();
}

Hello Sanlok Lee, first of all, thanks for your answer. I have tried it, but the behavior is not exactly what I need. The point is that the final processing of the results should be done in the same thread of the initial execution (the one printing "Starting parallel executions") — RLS, Jan 29 '19 at 08:07
@RLS, You can achieve that behavior by simply adding an [observeOn](http://reactivex.io/documentation/operators/observeon.html) right before `.subscribe` call. You can pass in a scheduler that runs on your default thread using `scheduler.from()`. I will update the answer with an example when I get back home. — Sanlok Lee, Jan 29 '19 at 18:55
Sanlok Lee, many thanks for your help. I achieved to the same result making the current thread to wait and calling to a method which notifies the current thread to continue once the observable is complete: `.doOnComplete(() -> threadNotifier(currentThread))`, but my option is much more "dirty" and would require more error management. — RLS, Jan 30 '19 at 11:38

score 2 · Answer 2 · answered Jan 31 '19 at 05:46

2

I'd suggest that you're not thinking reactively enough:

public Single<List<String>> parallelExecution(List<Integer> calls) {
  return Observable
      .fromIterable(calls)
      .flatMap(val -> Observable.fromCallable(() -> simpleAsync(val).toString())
                                .subscribeOn(Schedulers.io())
      .toList();
}

the .toList() will collect all the results and provide a single items when the flatMap completes
you want to use subscribeOn, not observeOn.
this would be simpler if simpleAsync returned a reactive object.
if you need to keep parallelExecution non-reactive, use something like blockingGet.

answered Jan 31 '19 at 05:46

Tassos Bassoukos

16,017
2
36
40

Hello Tassos, thank you for your answer. I have tried your comments but I am no so familiar with `RxJava` as for making it work. I have changed `simpleAsync` to return an `Observable` with the result, so now the `flatMap` is `.flatMap(val -> simpleAsync(val)).subscribeOn(Schedulers.io()).toList()`. I have also added the `blockingGet`, so right now I do not use any `subscibed` sentence, but the 5 "parallel" executions are performed in the same thread (not the principal), so they are not executed in parallel – RLS Feb 01 '19 at 11:14

score 0 · Answer 3 · answered Oct 14 '21 at 15:12

Thanks for the question and the solutions, based on these I'm write a more clear utility method to call when will necessary:

 public static Disposable parallelExecution(Runnable afterExecutionRunnable, Runnable... calls) {       
    return Observable.fromArray(calls)
            .flatMap(callable -> Completable.fromRunnable(callable).subscribeOn(Schedulers.io()).toObservable())
            .toList().subscribeOn(Schedulers.io())
            .subscribe(voids -> {
                if (afterExecutionRunnable != null) {
                    afterExecutionRunnable.run();
                }
            });
}

Example of invocation:

    RxJavaUtils.parallelExecution(() -> {
                //Here action when all tasks ends
            },               
            this::taskMethod1,
            this::taskMethod2,
          //sequential tasks 
            () -> {
                taskMethod3();
                taskMethod4ThatDependsOf3();
            },
            this::taskMethod5);

You can cancel the parallel execution of tasks calling .dispose() over disposable returned by the utility method

Execute parallel tasks with RxJava and wait until the last one of them is completed

3 Answers3