Scala - Execute arbitrary number of Futures sequentially but dependently

Question

I'm trying to figure out the neatest way to execute a series of Futures in sequence, where one Future's execution depends on the previous. I'm trying to do this for an arbitrary number of futures.

User case:

I have retrieved a number of Ids from my database.
I now need to retrieve some related data on a web service.
I want to stop once I've found a valid result.
I only care about the result that succeeded.

Executing these all in parallel and then parsing the collection of results returned isn't an option. I have to do one request at a time, and only execute the next request if the previous request returned no results.

The current solution is along these lines. Using foldLeft to execute the requests and then only evaluating the next future if the previous future meets some condition.

def dblFuture(i: Int) = { i * 2 }
val list = List(1,2,3,4,5)
val future = list.foldLeft(Future(0)) {
  (previousFuture, next) => {
    for {
      previousResult <- previousFuture
      nextFuture <- { if (previousResult <= 4) dblFuture(next) else previousFuture }
    } yield (nextFuture)
  }
}

The big downside of this is a) I keep processing all items even once i've got a result i'm happy with and b) once I've found the result I'm after, I keep evaluating the predicate. In this case it's a simple if, but in reality it could be more complicated.

I feel like I'm missing a far more elegant solution to this.

I'm confused by the use case because it doesn't seem like it has the sort of data flow dependency you're describing ("one Future's execution depends on the previous"), since you're only executing the next Future if the previous Future's result was *empty*. What am I missing? That is, does the next Future depend on the previous result in any way other than merely to decide whether it executes? — Chris Martin, Oct 29 '14 at 03:08
This is quite similar to http://stackoverflow.com/questions/26438991/is-there-sequential-future-find/26439838#26439838 (see my answer there) and http://stackoverflow.com/questions/26349318/how-to-invoke-a-method-again-and-again-until-it-returns-a-future-value-contain — Alexey Romanov, Oct 29 '14 at 06:04
@ChrisMartin The next futures execution depends not only on whether the previous Future succeeded, but also on the response. For example: If the previous future contains WSResponse with status 404, execute the next future, otherwise don't. — healsjnr, Oct 29 '14 at 23:29
@AlexeyRomanov thanks for point out the other issue. This very similar, only difference i'm working with a sequence of Futures as opposed to a single future executed many times. — healsjnr, Oct 29 '14 at 23:35
@healsjnr The stream happens to be filled with equivalent futures in that question, but the answer doesn't depend on that in any way :) — Alexey Romanov, Oct 30 '14 at 06:24

Michael Zajac · Accepted Answer · 2014-10-29T03:52:36.817

Looking at your example, it seems as though the previous result has no bearing on subsequent results, and instead what only matters is that the previous result satisfies some condition to prevent the next result from being computed. If that is the case, here is a recursive solution using filter and recoverWith.

def untilFirstSuccess[A, B](f: A => Future[B])(condition: B => Boolean)(list: List[A]): Future[B] = {
    list match {
        case head :: tail => f(head).filter(condition).recoverWith { case _: Throwable => untilFirstSuccess(f)(condition)(tail) }
        case Nil => Future.failed(new Exception("All failed.."))
    }
 }

filter will only be called when the Future has completed, and recoverWith will only be called if the Future has failed.

def dblFuture(i: Int): Future[Int] = Future { 
     println("Executing.. " + i)
     i * 2 
 }

val list = List(1, 2, 3, 4, 5)

scala> untilFirstSuccess(dblFuture)(_ > 6)(list)
Executing.. 1
Executing.. 2
Executing.. 3
Executing.. 4
res1: scala.concurrent.Future[Int] = scala.concurrent.impl.Promise$DefaultPromise@514f4e98

scala> res1.value
res2: Option[scala.util.Try[Int]] = Some(Success(8))

score 2 · Answer 2 · answered Oct 29 '14 at 04:30

Neatest way, and "true functional programming" is scalaz-stream ;) However you'll need to switch to scalaz.concurrent.Task from scala Future for abstraction for "future result". It's a bit different. Task is pure, and Future is "running computation", but they have a lot in common.

  import scalaz.concurrent.Task
  import scalaz.stream.Process

  def dblTask(i: Int) = Task {
    println(s"Executing task $i")
    i * 2
  }

  val list = Seq(1,2,3,4,5)

  val p: Process[Task, Int] = Process.emitAll(list)

  val result: Task[Option[Int]] =
    p.flatMap(i => Process.eval(dblTask(i))).takeWhile(_ < 10).runLast

  println(s"result = ${result.run}")

Result:

Executing task 1
Executing task 2
Executing task 3
Executing task 4
Executing task 5
result = Some(8)

if your computation is already scala Future, you can transform it to Task

implicit class Transformer[+T](fut: => SFuture[T]) {
  def toTask(implicit ec: scala.concurrent.ExecutionContext): Task[T] = {
    import scala.util.{Failure, Success}
    import scalaz.syntax.either._
    Task.async {
      register =>
        fut.onComplete {
          case Success(v) => register(v.right)
          case Failure(ex) => register(ex.left)
        }
    }
  }
}

Thanks Eugene, I like this solution a lot, but i'm not using ScalaZ at the moment was hoping for a solution using standard Scala future (hence why i marked @LimbSoup as the correct answer). Will look at SacalZ in more detail though. — healsjnr, Oct 29 '14 at 23:33

Scala - Execute arbitrary number of Futures sequentially but dependently

2 Answers2