I've tried to model the situation I have in a real code.
The use case is simple: There is:
Future {blocking {
// in there it calls nested futures
}}
More detailed (global Execution context is used as parent one, but there is was attemp to use separete one for children):
object ExContexts {
val main = scala.concurrent.ExecutionContext.Implicits.global
val ecChildren = scala.concurrent.ExecutionContext.fromExecutor(Executors.newFixedThreadPool(1))
}
object NestedFutures2 extends App {
import scala.concurrent.ExecutionContext.Implicits.global
val cores = Runtime.getRuntime.availableProcessors
val works = parentWork(ExContexts.main) // main EC
val result1: Seq[Seq[Future[String]]] = Await.result(Future.sequence(works), Duration.Inf)
println("parents are done their work")
val result2: Seq[String] = Await.result(Future.sequence(result1.flatten), Duration.Inf)
// ---
def parentWork(ex:ExecutionContext): Seq[Future[Seq[Future[String]]]] = {
val works: Seq[Future[Seq[Future[String]]]] =
(1 to cores * 2) map { x =>
Future {
blocking { // will create new thread/place if needed
val parentName = Thread.currentThread.getName
println("parent: " + parentName + " started an action")
val playFutureOutcomes: Seq[Future[String]] = (1 to 10) map {stuffId =>
childPlay(parentName = parentName)(ExContexts.ecChildren)
}
Thread.sleep(1000)
println(s"[${timeStamp()}] parent: " + parentName + " has finished the action")
playFutureOutcomes
}
}
}
works
}
def childPlay(parentName:String)(ex:ExecutionContext):Future[String] = {
Future {
Thread.sleep(2000) // two seconds play session
val threadName = Thread.currentThread.getName
// log
println("child: " + threadName + " of " + parentName + " parent")
Thread.currentThread.getName
}
}
def timeStamp(pattern:String = "ss:mm : hh"): String = {
val fmt = DateTimeFormat.forPattern(pattern)
fmt.print(DateTime.now)
}
}
From the output (5 of 8 parrent futures are done) I see that children make use of parents threads...:
[01:31 : 05] parent: ForkJoinPool-1-worker-5 has finished the action
[01:31 : 05] parent: ForkJoinPool-1-worker-11 has finished the action
[01:31 : 05] parent: ForkJoinPool-1-worker-1 has finished the action
[01:31 : 05] parent: ForkJoinPool-1-worker-3 has finished the action
[01:31 : 05] parent: ForkJoinPool-1-worker-7 has finished the action
child: ForkJoinPool-1-worker-13 of ForkJoinPool-1-worker-1 parent
child: ForkJoinPool-1-worker-5 of ForkJoinPool-1-worker-5 parent
child: ForkJoinPool-1-worker-1 of ForkJoinPool-1-worker-1 parent
child: ForkJoinPool-1-worker-11 of ForkJoinPool-1-worker-11 parent
child: ForkJoinPool-1-worker-3 of ForkJoinPool-1-worker-3 parent
child: ForkJoinPool-1-worker-7 of ForkJoinPool-1-worker-7 parent
child: ForkJoinPool-1-worker-13 of ForkJoinPool-1-worker-11 parent
...
I expect parents one to done their work in ~1 second as it was if we don't have childrent. Also I would expect to see two connecton pools instead of one.
Q: what would be the suggestion to achive it?
UPDATE: I guess it is all about Join-Pool based upon Implicit.glocal is built. All nested Furue will be joined to one pool?
// What (I think) I see is that if we have 'global' execution context as parent/main,
// then whatever we pass as execution context to the child Future still will use global context
// and still will be using ONE connection pool.
// And then therefore wll have performance problem if children do not use 'blocking' -
// parents will be stuck in cores-long connection pool
// So: Children will 'inherit' execution context, but not 'blocking'
// and as children connected/joined to the execution context the parents will not be able to do
// anything about it but wait for the free space in the pool (when child is done)
// new additional threads will not be creating for the next parent because (i guess)
// Global execution context will not see the reason of/for it.
// Like a snapshot of the moment of time:
// Connection Pool: [ch1, ch2, ch3, p1]
// It things:Ok there is one parent (p1) in the pool if one more parent
// future will come I create for it new thread it the pool.
// But because we have more joined children than parents in the list
// the moment when Global ex context can help parent will never come.
Should then I not to use that global context if I want parents to be independant on childrent and vice versa? (funny how many people in Internet are playing with global only.)
Say I know ma app has 20 000 clients and each of them makes 1-100 sec long operations (op1->op1.op1) all the time. Then I just could dedicate one executon context with 20 000 connecton pool (max). And as many execution context as needed for all sort of long-running operations (parent->child where child is not reportng to the parent). Say: printingExCon
, logExCon
, readStufExCon
, writeStuffExCon
But if I know that one oparation should include anoter one and they maybe joined then (only) I may use that global Ex context?
Otherwise if I would use global ex context as may man context - all and everyone would join to it and I would end up having performance problem that I explained at te beginning of this Sunday post..
UPDATE: I wasted good chunk of time trying to understand why fixed version (with 'ex' passed to childPlay's Future{}(ex) was producing an output that ignore sleeps and behaves weired on many other cases. The problem was fixed when instead of running/buildng in IntellyJIDEA 14 (by CTRL-SHIFT+F10) i build it wit sbt run..