Akka streams don't run when Source has large number of records

Question

I'm trying to write a very simple introductory example of using Akka Streams. I'm attempting to basically create a stream that takes a range of integers as a source and filters out all the integers that are not prime, producing a stream of prime integers as its output.

The class that constructs the stream is rather simple; for that I have the following.

import akka.NotUsed;
import akka.actor.ActorSystem;
import akka.stream.javadsl.Flow;
import com.aparapi.Kernel;
import com.aparapi.Range;

import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;

public class PrimeStream {
    private final AverageRepository averageRepository = new AverageRepository();
    private final ActorSystem actorSystem;

    public PrimeStream(ActorSystem actorSystem) {
        this.actorSystem = actorSystem;
    }

    public Flow<Integer, Integer, NotUsed> filterPrimes() {
        return Flow.of(Integer.class).grouped(10000).mapConcat(PrimeKernel::filterPrimes).filter( v -> v != 0);
    }
}

When I run the following test, it works fine.

private final ActorSystem actorSystem = ActorSystem.create("Sys");

@Test
public void testStreams() {
    Flow<Integer, Integer, NotUsed> filterStream = new PrimeStream(actorSystem).filterPrimes();
    Source<Integer, NotUsed> flow = Source.range(10000000, 10001000).via(filterStream);
    flow.runForeach(System.out::println, ActorMaterializer.create(actorSystem));
}

However, when I increase the range by a factor of x10 by changing the line in the test to the following, it no longer works.

Source<Integer, NotUsed> flow = Source.range(10000000, 10010000).via(filterStream);

Now when the test runs, no exceptions are thrown, no warnings. It simply runs, then exits, without displaying any text to the console at all.

Just to be extra certain that the problem wasn't in my primality test itself, I ran the test over the same range without using Akka Streams, and it runs fine. The following code runs without a problem.

@Test
public void testPlain() {
    List<Integer> in = IntStream.rangeClosed(10000000, 10010000).boxed().collect(Collectors.toList());
    List<Integer> out = PrimeKernel.filterPrimes(in);
    System.out.println(out);
}

Just for the sake of clarity, the primality test itself takes in a list of integers and sets any element in the list to 0 if it is not prime.

As suggested by @RamonJRomeroyVigil if i remove the mapConcat part all together but leave everythig the same it does, in fact, print out 10,000 integers. However If i leave everything the same but simply replace filterPrimes with a method that just returns the method parameter as is without touching it, then it doesnt print anything to the screen at all. I've also tried adding a println to the begining filterPrime to debug it. Whenever it doesnt print any output that includes the debugging statement. So no attempt is even made to call filterPrimes at all.

Just a wild guess, can you try to change the grouped value from 10000 to 1000? — Ankur, May 10 '18 at 04:28
@Ankur if i do that it does in face work But i want it to run in batches of 10,000 because that method will process the primes on a GPU acelerated framework. so will work best in large batches. — Jeffrey Phillips Freeman, May 10 '18 at 04:35
Wait a second, `runForeach` returns a future, so if you want to see all the numbers getting printed then you have to await on the future otherwise the test function returns and the program terminates without the future getting completed. — Ankur, May 10 '18 at 04:53
@Ankur perhaps the tutorial i pulled from didnt show that (or maybe i misunderstood something).. let me go try that... — Jeffrey Phillips Freeman, May 10 '18 at 05:00
@Ankur So runForEach isnt returning a Future for me, but it does return something which may behave somewhat similarly... so i think you may be on to something... — Jeffrey Phillips Freeman, May 10 '18 at 05:02
Yeah it return https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletionStage.html .. which represent async computation just like future. — Ankur, May 10 '18 at 05:04
@Ankur I tried calling .handle on it in an attempt to get it to complete but without success. Could you propose an answer to this question with a suggested solution so I can get a better idea how you think I could solve this. It will also let me upvote and accept the answer. — Jeffrey Phillips Freeman, May 10 '18 at 05:05

score 5 · Accepted Answer · answered May 10 '18 at 05:08

5

runForeach returns a CompletionStage, so if you want to see all the numbers getting printed then you have to await on the CompletionStage otherwise the test function returns and the program terminates without the CompletionStage getting completed.

Example:

flow.runForeach(System.out::println, ActorMaterializer.create(actorSystem)).toCompletableFuture().join();

answered May 10 '18 at 05:08

Ankur

33,367
2
46
72

This does appear to work now thanks. Oddly the `.thenRun(...)` method as suggested in the official docs (which I tried earlier) doesnt do the trick. Very confusing but there is probably something obvious im missing. Regardless im accepting this as the correct answer, thanks. – Jeffrey Phillips Freeman May 10 '18 at 05:15
`thenRun` is just saying that when the stage completes run the given function but it doesn't make the current thread to wait for the async computation to complete, instead it returns a new `CompletionStage` that again you will have to await for. – Ankur May 10 '18 at 05:23
Thanks, this tutorial seems to leave that part out then, as fif a few others. Very frustrating: https://doc.akka.io/docs/akka/2.5/stream/stream-quickstart.html – Jeffrey Phillips Freeman May 10 '18 at 05:28

Akka streams don't run when Source has large number of records

1 Answers1