How do you deal with futures in Akka Flow?

Question

I have built an akka graph that defines a flow. My objective is to reformat my future response and save it to a file. The flow can be outlined bellow:

val g = RunnableGraph.fromGraph(GraphDSL.create() { implicit builder: GraphDSL.Builder[NotUsed] =>
      import GraphDSL.Implicits._
      val balancer = builder.add(Balance[(HttpRequest, String)](6, waitForAllDownstreams = false))
      val merger = builder.add(Merge[Future[Map[String, String]]](6))
      val fileSink = FileIO.toPath(outputPath, options)
      val ignoreSink = Sink.ignore
      val in = Source(seeds)
      in ~> balancer.in
      for (i <- Range(0,6)) {
        balancer.out(i) ~>
          wikiFlow.async ~>
          // This maps to a Future[Map[String, String]]
          Flow[(Try[HttpResponse], String)].map(parseHtml) ~>
          merger
      }

      merger.out ~>
      // When we merge we need to map our Map to a file
      Flow[Future[Map[String, String]]].map((d) => {
        // What is the proper way of serializing future map
        // so I can work with it like a normal stream into fileSink?

        // I could manually do ->
        // d.foreach(someWriteToFileProcess(_))
        // with ignoreSink, but this defeats the nice
        // akka flow
      }) ~>
      fileSink

      ClosedShape
    })

I can hack this workflow to write my future map to a file via foreach, but I'm afraid this could somehow lead to concurrency issues with FileIO and it just doesn't feel right. What is the proper way to handle futures with our akka flow?

Possible duplicate of [Create backpressure from a Future inside an Akka stream](http://stackoverflow.com/questions/39909303/create-backpressure-from-a-future-inside-an-akka-stream) — Ramón J Romero y Vigil, Feb 13 '17 at 18:22

sarveshseri · Accepted Answer · 2017-02-06T13:05:43.450

26

The easiest way to create a Flow which involves an asynchronous computation is by using mapAsync.

So... lets say you want to create a Flow which consumes Int and produces String using an asynchronous computation mapper: Int => Future[String] with a parallelism of 5.

val mapper: Int => Future[String] = (i: Int) => Future(i.toString)

val yourFlow = Flow[Int].mapAsync[String](5)(mapper)

Now, you can use this flow in your graph however you want.

An example usage will be,

val graph = GraphDSL.create() { implicit builder =>
  import GraphDSL.Implicits._

  val intSource = Source(1 to 10)

  val printSink = Sink.foreach[String](s => println(s))

  val yourMapper: Int => Future[String] = (i: Int) => Future(i.toString)

  val yourFlow = Flow[Int].mapAsync[String](2)(yourMapper)

  intSource ~> yourFlow ~> printSink

  ClosedShape
}

edited Feb 06 '17 at 13:05

answered Feb 06 '17 at 12:58

sarveshseri

13,738
28
47

I'll check this out later tonight. Didn't know about mapAsync, I didn't see it in their guide. Thanks! – Dr.Knowitall Feb 06 '17 at 16:41
Now the question becomes what happens when that future nests other futures, how do you manage the thread context of that. – PlexQ Mar 05 '19 at 02:41
@PlexQ That depends on the nature of these "nested" Futures. Since the stream progression is dependent only on the Top layer Future, you are free to deal with nested futures anyway you want. But you almost (other than for situational exceptions) never want to use the same execution context for these nested futures. – sarveshseri Mar 05 '19 at 05:57

How do you deal with futures in Akka Flow?

1 Answers1