0

I am specifically using twitter's AsyncStream and I need to take the results of concurrent processing, and make it into a Seq, but the only way I've managed to get working is horrendous. This feels like it should be a one liner, but all my attempts with Await and force have either hung or not processed the work.

Here's what I have working - what's a more idiomatic way of doing this?

  def processWork(work: AsyncStream[Work]): Seq[Result] = {
    // TODO: make less stupid
    val resultStream = work.flatMap { processWork }
    var results : Seq[Result] = Nil
    resultStream.foreach {
      result => {
        results = results :+ result
      }
    }
    results
  }
L0ne
  • 470
  • 3
  • 10
  • `foreach` is an `action` which will bring all of your data in `driver node` for processing. Why don't you use `lazy transformation` `map` instead of `foreach`? – Ramesh Maharjan May 13 '17 at 00:55
  • 1
    is `Await.result(work.toSeq(), 1.second)` hanging on you with the imports `import com.twitter.conversions.time._` and `import com.twitter.util.Await`? – Matt Fowler May 13 '17 at 00:58
  • @RameshMaharjan When I use `map` instead of `foreach` (which doesn't actually clean this up because I still have to load it into `var results: Seq[Result]`, I seem to be only getting the first result before it returns – L0ne May 13 '17 at 15:55
  • @MattFowler Trying `Await.result(resultStream.toSeq(), 1.second)` just times out (it does with longer durations, too) - it's never getting the results but it hang for the duration of the timeout - and yes, those are the libraries I'm using – L0ne May 13 '17 at 15:56
  • @L0ne, I am guessing `mapWithState` should be the best choice to update `results: Seq[Result]` with every `stream`. – Ramesh Maharjan May 13 '17 at 16:04
  • 1
    like @MattFowler mentioned - `Await.result(work.toSeq(), 1.second)` seems like the way to go but you say it hangs - are you sure the stream is finite? If it is infinite hanging on waiting for it's values makes sense. – Dani May 14 '17 at 15:44
  • @Dani Nailed it, thanks. The stream was never terminating – L0ne May 15 '17 at 17:32
  • @L0ne Cool, happy it worked :) Posted the solution as an answer as well. – Dani May 15 '17 at 18:04

1 Answers1

1

Like @MattFowler pointed out - you can force the stream and wait until it's complete using:

Await.result(resultStream.toSeq, 1.second)

toSeq will start realizing the stream and return a Future[Seq[A]] that completes once all the element are resolved, as documentated here.

You can then block until the future completes by using Await.result.

Make sure your stream is finite! Calling Await.result(resultStream.toSeq, 1.second) would hang forever.

Dani
  • 1,012
  • 9
  • 15
  • Ultimately, I fixed the stream to be finite, and then `Await.result(resultStream.toSeq)` worked fine – L0ne May 18 '17 at 18:11