0

I want to run a spark job locally for testing. If spark-submit and a assembled jar are used it works just fine.

However if sbt run is used I get a very strange error https://gist.github.com/geoHeil/946dd7706f44f338101c8332f4e13c1a

Trying to set java-opts like

javaOptions ++= Seq("-Xms512M", "-Xmx2048M", "-XX:MaxPermSize=2048M", "-XX:+CMSClassUnloadingEnabled")

Did not help to solve the problem.

Trying to fiddle with memory settings in local[*] mode like

.set("spark.executor.memory", "7g")
.set("spark.driver.memory", "7g")

did only spawn further problems of ExecutorLostFailure

Georg Heiler
  • 16,916
  • 36
  • 162
  • 292

1 Answers1

2

I never ran into this issue specifically but I think spark code is not made to be run with sbt run. I even remember reading about it in the docs but could not find it as of now.

I guess what you should do instead is compiling it with sbt and use spark-submit instead.

OberstK
  • 140
  • 6
  • But when a spark-job is embedded via an actor in a play-framework application how would you start it in such a context? – Georg Heiler May 11 '16 at 20:21
  • I have no experience with the play framework unfortunately. Can not be of help here. – OberstK May 13 '16 at 11:38
  • Or with akka. The point I want to make is: it is not a single one-off spark job but integrated into an scala application. – Georg Heiler May 13 '16 at 11:50
  • You could try using something like the sparklauncher which is used to start spark jobs from within Java or Scala code. An example can be found here: https://blog.knoldus.com/2015/06/26/startdeploy-apache-spark-application-programmatically-using-spark-launcher/ – OberstK May 14 '16 at 22:41
  • Very interesting @OberstK however, if I understand correctly sparkLauncher still requires to assemble the jar. And then a spark-submit is equally simple (for testing purposes) – Georg Heiler May 15 '16 at 08:38
  • Absolutely. For testing purposes spark-submit seems the way to go. Only other way may be using the spark shell on its own but that one is pretty limited. – OberstK May 18 '16 at 12:19