3

I am new to Spark and need help with the error: java.lang.NoClassDefFoundError: org/apache/spark/rdd/RDD$

I am creating a standalone Spark example in Scala. I ran sbt clean package and sbt assembly to package the scala spark code. Both completed successfully without any error. Any operation on a RDD throws error. Any pointers to fix this issue will be really helpful.

I invoke the job using spark-submit command.

$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.GroupTest /Users/../spark_workspace/spark/examples/target/scala-2.10/spark-examples_2.10-1.3.0-SNAPSHOT.jar

Adam Yost
  • 3,616
  • 23
  • 36
vikrame
  • 485
  • 2
  • 12
  • Did you download a binary distribution? I so, which one. Are there other suggestive messages written to the console? is $SPARK_HOME the same as /Users/../spark_workspace/spark/? – Dean Wampler Feb 02 '15 at 20:23
  • Hi, I am having same exactly problem. How did you solved your problem? – Shekhar Mar 29 '15 at 18:10
  • Do you compile and run your application with different spark-cores? – zergood Jul 10 '15 at 10:07

1 Answers1

0

I managed to throw this error and get past it. This is definitely a YMMV answer, but I leave it here in case it eventually helps someone.

In my case, I was running a homebrew installed spark (1.2.0) and mahout (0.11.0) on a mac. It was pretty baffling to me because if I ran a mahout command line by hand I didn't get the error, but if I invoked it from within some python code it threw the error.

I realized that I had updated my SPARK_HOME variable in my profile to use 1.4.1 instead and had re-sourced it in my by-hand terminal. The terminal where I was running the python code was still using 1.2.0. I re-sourced my profile in my python terminal and now it "just works."

The whole thing feels very black magicky, if I were to guess some rational reason for this error being thrown, maybe it's because one moving part assumes a different spark version, architecture, whatever than you have. That seems to be the solution hinted at in the comments, too.

Foobie Bletch
  • 300
  • 2
  • 8