NoClassDefFoundError: Could not initialize XXX class after deploying on spark standalone cluster

Question

I wrote a spark streaming application built with sbt. It works perfectly fine locally, but after deploying on the cluster, it complains about a class I wrote which clearly in the fat jar (checked using jar tvf). The following is my project structure. XXX object is the one that spark complains about

src
`-- main
    `-- scala
        |-- packageName
        |   `-- XXX object
        `-- mainMethodEntryObject

My submit command:

$SPARK_HOME/bin/spark-submit \
  --class mainMethodEntryObject \
  --master REST_URL\
  --deploy-mode cluster \
  hdfs:///FAT_JAR_PRODUCED_BY_SBT_ASSEMBLY

Specific error message:

java.lang.NoClassDefFoundError: Could not initialize class XXX

can you add the command used to launch job and when you run `jar tvf `what response it returns? — Rahul Sharma, Apr 26 '17 at 03:22
@techsquids yes, I run `jar tvf target/scala-2.11/FAT_JAR_NAME` at root of my project, and it has `packageName/XXX.class` in it — Dr.Pro, Apr 26 '17 at 03:29
strange, normally all of your jar classes should be available at runtime. try adding main jar in --lib also in `spark-submit` command. — Rahul Sharma, Apr 26 '17 at 03:44
can it be a java installed version problem? check if all the machines are running with the same java version. I had similar problems previously due to that — pedrorijo91, Apr 26 '17 at 08:56

score 4 · Answer 1 · answered May 29 '18 at 17:25

I ran into this issue for a reason similar to this user: http://apache-spark-developers-list.1001551.n3.nabble.com/java-lang-NoClassDefFoundError-is-this-a-bug-td18972.html

I was calling a method on an object that had a few variables defined on the object itself, including spark and a logger, like this

val spark = SparkSession
  .builder()
  .getOrCreate()

val logger = LoggerFactory.getLogger(this.getClass.getName)

The function I was calling called another function on the object, which called another function, which called yet another function on the object inside of a flatMap call on an rdd.

I was getting the NoClassDefFoundError error in a stacktrace where the previous 2 function calls in the stack trace were functions on the class Spark was telling me did not exist.

Based on the conversation linked above, my hypothesis was that the global spark reference wasn't getting initialized by the time the function that used it was getting called (the one that resulted in the NoClassDefFoundError exception).

After quite a few experiments, I found that this pattern worked to resolve the problem.

// Move global definitions here
object MyClassGlobalDef {

  val spark = SparkSession
    .builder()
    .getOrCreate()

  val logger = LoggerFactory.getLogger(this.getClass.getName)

}

// Force the globals object to be initialized
import MyClassGlobalDef._

object MyClass {
  // Functions here
}

It's kind of ugly, but Spark seems to like it.

i had a similar issue and this solved it. but it's still puzzling to me. my code used to run fine until a couple of days ago. do you know what's root cause for this? — qkhhly, May 31 '20 at 01:19
I suspect the problem has something to do with how Spark does serialization, but aside from that I'm not sure. I haven't dug into this deep enough to debug. — turtlemonvh, Jun 15 '20 at 14:03

score 2 · Answer 2 · answered Dec 08 '17 at 20:51

It's difficult to say without the code but it looks like a problem of serialization of your XXX object. I can't say I'm understand perfectly why, but the point is that the object is not shipped to the executor.

The solution that worked for me is to convert your object to a class that extends Serializable and just instantiate it where you need it. So basically, if I'm not wrong you have

object test {
   def foo = ...
}

which would be used as test.foo in your main, but you need at minimum

class Test extends Serializable {
   def foo = ...
}

and then in your main have val test = new Test at the beginning and that's it.

score 1 · Answer 3 · answered Jan 12 '21 at 09:06

1

It is related to serialization. I fixed this by adding "implements Serializable" and serialVersionUID field to given class.

answered Jan 12 '21 at 09:06

Mariusz Tasarz

11
1

This was the fix for me! – tjheslin1 Jan 12 '21 at 17:57

NoClassDefFoundError: Could not initialize XXX class after deploying on spark standalone cluster

3 Answers3