0

I am trying to use the spark job server on a cluster on CDH 5.11, spark version 1.6.0 When I try to start the spark jobserver on the deployed machine I get this error log

[ERROR] [06/02/2017 15:30:14.966] [JobServer-akka.actor.default-dispatcher-3] [ActorSystem(JobServer)] Uncaught error from thread [JobServer-akka.actor.default-dispatcher-3] shutting down JVM since 'akka.jvm-ex$
java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
at akka.event.slf4j.Logger$.apply(Slf4jLogger.scala:30)
at akka.event.slf4j.SLF4JLogging$class.log(Slf4jLogger.scala:19)
at akka.event.slf4j.Slf4jLogger.log$lzycompute(Slf4jLogger.scala:54)
at akka.event.slf4j.Slf4jLogger.log(Slf4jLogger.scala:54)
at akka.event.slf4j.Slf4jLogger$$anonfun$receive$1.applyOrElse(Slf4jLogger.scala:80)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.ClassNotFoundException: org.slf4j.LoggerFactory
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 14 more

Uncaught error from thread [JobServer-akka.actor.default-dispatcher-3] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[JobServer]
java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
at akka.event.slf4j.Logger$.apply(Slf4jLogger.scala:30)
at akka.event.slf4j.SLF4JLogging$class.log(Slf4jLogger.scala:19)
at akka.event.slf4j.Slf4jLogger.log$lzycompute(Slf4jLogger.scala:54)
at akka.event.slf4j.Slf4jLogger.log(Slf4jLogger.scala:54)
at akka.event.slf4j.Slf4jLogger$$anonfun$receive$1.applyOrElse(Slf4jLogger.scala:80)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.ClassNotFoundException: org.slf4j.LoggerFactory
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 14 more

Did anybody faced the same issue or provide some help ?

Reuben Borrison
  • 1

1 Answers1

0

Few things here!!

How are you deploying? The problem seems to be that some how the SLF4J jar is excluded from SJS class path. It is possible that there is a version conflict on it.

And check the spark dependencies in the UI. (and find SLF4J there!!)

You have to check scala version; spark version and the Spark-Job-Server versions if they're conflicting!!!

In your case, for spark version 1.6.0; the selected scala version should be 2.10.4. (not any other versions like 2.11 or 2.12)

Raktotpal Bordoloi
  • 1,009
  • 8
  • 15
  • Thanks for the comment but the problem already got solved! It was the same error as mentioned by you but in addition to it there was improper classpath set in CDH as I did an upgrade which didn't update the spark classpath.txt. After updating it everthing went fine. Also Now I use a different repo for spark job server ([bjoern's repo](https://github.com/bjoernlohrmann/spark-jobserver) which is built for several CDH versions! – Reuben Borrison Jun 08 '17 at 11:59