Scala Observable Creation Blocks My Futures

Question

I am wanting to process ea. query fetch (potentially multiple fetches per query) asynchronously. In order to do this, I pass the processing function (which returns a Future) to my query method to call it for ea. fetch. I don't know beforehand the result size of my query; I only know the max size of my fetch. Therefore, my query returns an Observable (as opposed to a List for ex. where I need to know the size beforehand). The only problem is that when I use Observable create or apply, it will internally block until my Future is completed before it calls the next onNext -- effectively, removing an performance gains I was hoping to get from the futures. The Observable from factory method does not block but it takes an Iterable. I can pass it a mutable Iterable and grow as new fetches come in. Somebody have a more referentially transparent sol'n? Here's the code:

object Repository {
  def query(fetchSize: Int)(f: Set[Int] => Future[Set[Int]]): Observable[Future[Set[Int]]] = {
    // observable (as opposed to list) because modeling a process 
    // where the total result size is unknown beforehand. 
    // Also, not creating or applying because it blocks the futures
    val mut = scala.collection.mutable.Set[Future[Set[Int]]]()
    val obs = Observable.from(mut)
    1 to 2100 by fetchSize foreach { i =>
      mut += f(DataSource.fetch(i, fetchSize))
    }
    obs
  }
}

Producing Futures with Observable has no sence for me. I think this is completely wrong — Nyavro, Nov 18 '15 at 07:30
So since calling `f` will run `DataSource.fetch` before entering `f`, the `1 to 2100 by fetchSize foreach` statement will be blocking as well. It will run `DataSource.fetch` one by one. — zsxwing, Nov 19 '15 at 00:09
@zsxwing, yes this is by design to model inherent jdbc interactions. My intention is to wait until **each** fetch is done but **not** **all** the fetches before starting the `f` computations. Is this making sense? — juanchito, Nov 19 '15 at 14:43
`def process(): List[Int] = { for { oddsFuture <- oddObs } yield Await.result(oddsFuture, Inf) }.toBlocking.toList.flatten`. Context [here](https://gist.github.com/mayonesa/a0f808c6c6f585a37155) — juanchito, Nov 19 '15 at 22:14
@zsxwing, regarding the previous comment's context [link](https://gist.github.com/mayonesa/a0f808c6c6f585a37155), any code critique is greatly appreciated. ;) — juanchito, Nov 19 '15 at 22:24
As it turns out, @Nyavro was on the right track. I realized that you don't need to know the size of the result set beforehand because you're not waiting to return the final result set before you start triggering the futures and their callbacks. I scrapped rx-scala completely and went w/ `Set` instead of `Observable`: code [here](https://gist.github.com/mayonesa/a0f808c6c6f585a37155). — juanchito, Nov 23 '15 at 19:20

score 0 · Accepted Answer · answered Nov 22 '15 at 00:52

0

I was able to remove mutability by using foldLeft:

(1 to 21 by fetchSize).foldLeft(Observable just Future((Set[Int]()))) { (obs, i) =>
  obs + f(DataSource.fetch(i)())
}

where:

implicit class FutureObservable(obs: Observable[Future[Set[Int]]]) {
  def +(future: Future[Set[Int]]) =
  obs merge (Observable just future)
}

The only thing is that I don't like what I had to do create an empty Observable that the compiler didn't gripe about. If anyone has a better answer, please post a it and I will mark it.

answered Nov 22 '15 at 00:52

juanchito

493
1
5
16

you could use `reduceLeft` it will take it's initial accumulator as first element of collection. If there is none you will get exception. Solution for this is `reduceLeftOption`. – Łukasz Nov 22 '15 at 10:43
thanks @Łukasz. I realized that I didn't need to use `Observable` and went w/ `Set` instead. Here's the final [code](https://gist.github.com/mayonesa/a0f808c6c6f585a37155) – juanchito Nov 23 '15 at 19:24

Scala Observable Creation Blocks My Futures

1 Answers1