Communication with Spark using Spark JobServer in docker

Question

I have launched spark cluster in single mode.

start-master.sh -h 10.0.0.56
start-slave.sh spark://10.0.0.56:7077

I can successfully run job using spark-core lib for Scala. I want to use Spark JobServer for jobs managing. I started it in docker on the master host node:

docker run -d -p 8090:8090 -e SPARK_MASTER=spark://10.0.0.56:7077 velvia/spark-jobserver:0.6.0

Then I try execute already added job (I verified through UI that it was added):

curl-d "input.string =somedata" '10.0.0.56:8090/jobs?appName=myjob&classPath=sparkscala.Job'

Response:

{
  "status": "ERROR",
  "result": {
    "message": "Futures timed out after [15 seconds]",
    "errorClass": "java.util.concurrent.TimeoutException",
    "stack": ["scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)", "scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)", "scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)", "akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread$$anon$3.block(ThreadPoo
lBuilder.scala:169)", "scala.concurrent.forkjoin.ForkJoinPool.managedBlock(ForkJoinPool.java:3640)", "akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread.blockOn(ThreadPoolBuilder.scala:167)", "akka.dispatch.BatchingExecutor$Batch.blockOn(BatchingExecutor.scala:101)", "scala.concurrent.Await$.result(packa
ge.scala:107)", "spark.jobserver.WebApi.spark$jobserver$WebApi$$getJobManagerForContext(WebApi.scala:478)", "spark.jobserver.WebApi$$anonfun$jobRoutes$1$$anonfun$apply$33$$anonfun$apply$34.apply(WebApi.scala:419)", "spark.jobserver.WebApi$$anonfun$jobRoutes$1$$anonfun$apply$33$$anonfun$apply$34.apply(WebApi.scala:412
)", "spray.routing.ApplyConverterInstances$$anon$18$$anonfun$apply$5.apply(ApplyConverterInstances.scala:53)", "spray.routing.ApplyConverterInstances$$anon$18$$anonfun$apply$5.apply(ApplyConverterInstances.scala:52)", "spray.routing.ConjunctionMagnet$$anon$1$$anon$2$$anonfun$happly$1$$anonfun$apply$1.apply(Directive.
scala:38)", "spray.routing.ConjunctionMagnet$$anon$1$$anon$2$$anonfun$happly$1$$anonfun$apply$1.apply(Directive.scala:37)", "spray.routing.directives.BasicDirectives$$anon$2.happly(BasicDirectives.scala:79)", "spray.routing.Directive$$anon$7$$anonfun$happly$4.apply(Directive.scala:86)", "spray.routing.Directive$$anon
$7$$anonfun$happly$4.apply(Directive.scala:86)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly
$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(Bas
icDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives
.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anonfun$mapRequestContext$1$$anonfun$apply$1.apply(BasicDirectives.scala:30)", "spray.routing.directives.BasicDirectives$$anonfun$mapRequestContext$1$$anonfun$a
pply$1.apply(BasicDirectives.scala:30)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anonfun$mapRequestContext$1$$a
nonfun$apply$1.apply(BasicDirectives.scala:30)", "spray.routing.directives.BasicDirectives$$anonfun$mapRequestContext$1$$anonfun$apply$1.apply(BasicDirectives.scala:30)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$an
on$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:32)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:31)", "spray.rout
ing.RequestContext$$anonfun$withRejectionHandling$1.applyOrElse(RequestContext.scala:130)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)", "scala.runtime.AbstractPartialFunction$m
cVL$sp.apply(AbstractPartialFunction.scala:25)", "spray.routing.RequestContext$$anon$1.handle(RequestContext.scala:84)", "akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)", "spray.routing.RequestContext$$anon$3.handle(RequestContext.scala:102)", "akka.spray.UnregisteredActorRefBase.$bang(U
nregisteredActorRefBase.scala:72)", "spray.routing.RequestContext.reject(RequestContext.scala:202)", "spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:35)", "spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:34)", "spray.routing.directives.BasicDirectives$$anon$3
$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:32)", "spray.routing.RouteConcatenation$Rout
eConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:31)", "spray.routing.RequestContext$$anonfun$withRejectionHandling$1.applyOrElse(RequestContext.scala:130)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)", "scala.runtime.AbstractPartialF
unction$mcVL$sp.apply(AbstractPartialFunction.scala:33)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)", "spray.routing.RequestContext$$anon$1.handle(RequestContext.scala:84)", "akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)", "spray.routing.Requ
estContext$$anon$3.handle(RequestContext.scala:102)", "akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)", "spray.routing.RequestContext.reject(RequestContext.scala:202)", "spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:35)", "spray.routing.directives.RouteDire
ctives$$anon$1.apply(RouteDirectives.scala:34)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$ti
lde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:32)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:31)", "spray.routing.RequestContext$$anonfun$withRejectionHandling$1.applyOrElse(RequestContext.scala:130)", "scala.runtime.AbstractPartial
Function$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)", "spray.routing.RequestContext$$anon$1.handle(RequestContext.scala:84)", "
akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)", "spray.routing.RequestContext$$anon$3.handle(RequestContext.scala:102)", "akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)", "spray.routing.RequestContext.reject(RequestContext.scala:202)", "spray.routing.direct
ives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:35)", "spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:34)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.a
pply(BasicDirectives.scala:92)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:32)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:31)", "spray.routing.RequestContext$$anon
fun$withRejectionHandling$1.applyOrElse(RequestContext.scala:130)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)", "scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPar
tialFunction.scala:25)", "spray.routing.RequestContext$$anon$1.handle(RequestContext.scala:84)", "akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72)", "spray.routing.RequestContext.reject(RequestContext.scala:202)", "spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scal
a:35)", "spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:34)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92)", "spray.routing.RouteCo
ncatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:30)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:29)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:30)", "spray.routing.R
outeConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:29)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:30)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:29)", "spray.rou
ting.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:30)", "spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1.apply(RouteConcatenation.scala:29)", "spray.routing.directives.OnSuccessFutureMagnet$$anon$2$$anonfun$happly$2$$anonfun$apply$2.apply(FutureDirective
s.scala:81)", "spray.routing.directives.OnSuccessFutureMagnet$$anon$2$$anonfun$happly$2$$anonfun$apply$2.apply(FutureDirectives.scala:79)", "scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)", "akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)", "akka.dispatch.Ba
tchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)", "akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)", "akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)", "scala.concurrent.BlockContext$.withBlockContext(BlockContext.sc
ala:72)", "akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)", "akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)", "akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)", "scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)", "s
cala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)", "scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)", "scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)"]
  }

Logs on Job Server:

[2016-05-05 13:27:00,655] INFO  ient.AppClient$ClientActor [] [] - Connecting to master akka.tcp://sparkMaster@10.0.0.56:7077/user/Master...
[2016-05-05 13:27:14,457] ERROR ocalContextSupervisorActor [] [] - Exception after sending Initialize to JobManagerActor
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://JobServer/user/context-supervisor/c7f81d9f-sparkscala.Server#-306023562]] after [15000 ms]
    at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:333)
    at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)
    at scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694)
    at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:691)
    at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467)
    at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419)
    at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423)
    at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
    at java.lang.Thread.run(Thread.java:745)

Spark version: 1.6.1 Prebuilt for Hadoop 2.6
Spark JobServer: 0.6.0
Job Scala version 2.10.6

score 0 · Accepted Answer · answered May 06 '16 at 12:13

0

I updated to las version of jobserver (0.6.2.mesos-0.28.1.spark-1.6.1) and all is good now. May be was collision with spark versions.

answered May 06 '16 at 12:13

Cortwave

4,747
2
25
42

Will this work if sjs is hosted outside master host node? I have this scenarios and spark worker is connecting back to docker container IP which is not reachable. – noorul May 12 '16 at 07:28
It should work remotely, you should just bind ports and specify correct external ips in all configs – Cortwave May 12 '16 at 09:13
when I specify docker host ip in spark.driver.host, job fails to start because it tries to bind to a random port on docker host ip. – noorul May 12 '16 at 10:06
Do you specified port number? – Cortwave May 12 '16 at 10:16
It picked random port. But the issue is that a process can only bind to container ip not the host ip. spark.driver.host is also used for binding. – noorul May 12 '16 at 11:04

Communication with Spark using Spark JobServer in docker

1 Answers1