1

Getting java.lang.OutOfMemoryError: Java heap space in Spark job server logs and the job server goes down :

[2017-06-01 19:09:26,708] ERROR akka.actor.ActorSystemImpl [] [ActorSystem(JobServer)] - Uncaught error from thread [JobServer-akka.actor.default-dispatcher-18] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled

java.lang.OutOfMemoryError: Java heap space at spray.http.HttpData$NonEmpty.toByteArray(HttpData.scala:209) at spray.httpx.unmarshalling.BasicUnmarshallers$$anon$1.apply(BasicUnmarshallers.scala:30) at spray.httpx.unmarshalling.BasicUnmarshallers$$anon$1.apply(BasicUnmarshallers.scala:29) at spray.httpx.unmarshalling.UnmarshallerLifting$$anon$3.apply(UnmarshallerLifting.scala:35) at spray.httpx.unmarshalling.UnmarshallerLifting$$anon$3.apply(UnmarshallerLifting.scala:34) at spray.httpx.unmarshalling.UnmarshallerLifting$$anon$1.apply(UnmarshallerLifting.scala:25) at spray.httpx.unmarshalling.UnmarshallerLifting$$anon$1.apply(UnmarshallerLifting.scala:24) at spray.httpx.unmarshalling.package$PimpedHttpRequest.as(package.scala:47) at spray.routing.directives.MarshallingDirectives$$anonfun$entity$1.apply(MarshallingDirectives.scala:35) at spray.routing.directives.MarshallingDirectives$$anonfun$entity$1.apply(MarshallingDirectives.scala:35) at spray.routing.directives.BasicDirectives$$anonfun$extract$1.apply(BasicDirectives.scala:86) at spray.routing.directives.BasicDirectives$$anonfun$extract$1.apply(BasicDirectives.scala:86) at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92) at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92) at spray.routing.directives.BasicDirectives$$anonfun$mapRequestContext$1$$anonfun$apply$1.apply(BasicDirectives.scala:30) at spray.routing.directives.BasicDirectives$$anonfun$mapRequestContext$1$$anonfun$apply$1.apply(BasicDirectives.scala:30) at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92) at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92) at spray.routing.directives.BasicDirectives$$anonfun$mapRequestContext$1$$anonfun$apply$1.apply(BasicDirectives.scala:30) at spray.routing.directives.BasicDirectives$$anonfun$mapRequestContext$1$$anonfun$apply$1.apply(BasicDirectives.scala:30) at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92) at spray.routing.directives.BasicDirectives$$anon$3$$anonfun$happly$1.apply(BasicDirectives.scala:92) at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:32) at spray.routing.RouteConcatenation$RouteConcatenation$$anonfun$$tilde$1$$anonfun$apply$1.apply(RouteConcatenation.scala:31) at spray.routing.RequestContext$$anonfun$withRejectionHandling$1.applyOrElse(RequestContext.scala:130) at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33) at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33) at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25) at spray.routing.RequestContext$$anon$1.handle(RequestContext.scala:84) at akka.spray.UnregisteredActorRefBase.$bang(UnregisteredActorRefBase.scala:72) at spray.routing.RequestContext.reject(RequestContext.scala:202) at spray.routing.directives.RouteDirectives$$anon$1.apply(RouteDirectives.scala:35)

How can i increase its memory?

User_qwerty
  • 375
  • 1
  • 2
  • 10
  • try increasing driver memory by using --driver-memory 8g (or whatever value works for you) – tbone Jun 01 '17 at 17:01
  • Is it going down after a long run or running a specific program ? what is your current memory allocated to spark job server ? – Knight71 Jun 01 '17 at 19:19
  • It goes down on uploading jar, the jar size is around 350 MB. – User_qwerty Jun 02 '17 at 07:06
  • Also it says shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled. – User_qwerty Jun 02 '17 at 07:18
  • So i can disable 'akka.jvm-exit-on-fatal-error', but how can i pass memory parameter while uploading the jar to fix heap size error , also i am using java client api for this. – User_qwerty Jun 02 '17 at 07:22

0 Answers0