2

I am testing to load data from a csv to spark then save it in Elasticsearch but I am having some trouble on saving my RDD collection in Elasticsearch using spark. This error is raised when submitting job:

Exception in thread "main" java.lang.NoClassDefFoundError: org/elasticsearch/spark/rdd/api/java/JavaEsSpark

But my dependencies should be correct since I compiled with Maven...

My pom.xml is here : http://pastebin.com/b71KL903 .

The error is raised when I reach this line:

JavaEsSpark.saveToEs(javaRDD, "index/logements");

Rest of my code is here: http://pastebin.com/8yuJB68A

I have already searched about this problem but didn't find anything: https://discuss.elastic.co/t/problem-between-spark-and-elasticsearch/51942 .

https://github.com/elastic/elasticsearch-hadoop/issues/713 .

https://github.com/elastic/elasticsearch-hadoop/issues/585 .

I just learnt that : The "ClassNotFoundException" appears because Spark will shutdown its job classloader immediately in case of an exception so any other classes that need to be loaded, will fail causing the initial error to be hidden.

But I don't know how to proceed. I submitted my job with the verbose mode, but didn't see anything else: http://pastebin.com/j6zmyjFr

Thanks for your further help :)

kulssaka
  • 226
  • 8
  • 27

1 Answers1

3

Spark has executors and driver process. Executor runs in different node apart from driver node. Spark computes the rdd graph in various stages depending up on the transformations. And these stages have tasks that is executed on executors. So you need to pass the dependent jars to both executors and driver if you are using the library methods to compute rdd.

You should pass the dependent jars in --jars options in spark-submit

    spark-submit --jars $JARS \
     --driver-class-path $JARS_COLON_SEP \
     --class $CLASS_NAME $APP_JAR

In your case it would be

    spark-submit --jars elasticsearch-hadoop-2.3.2.jar \
    --master local[4]\
     --driver-class-path elasticsearch-hadoop-2.3.2.jar \
     --class "SimpleApp" target/simple-project-1.0.jar
Knight71
  • 2,927
  • 5
  • 37
  • 63
  • I don't know what driver-class-path I need and what class. I added the jar elasticsearch-hadoop but i don't know what to add after – kulssaka Jun 30 '16 at 09:48
  • bin/spark-submit --verbose --class "SimpleApp" --master local[4] target/simple-project-1.0.jar --jars elasticsearch-hadoop-2.3.2.jar and then what driver class i need to add ? the class it seems missing is JavaEsSpark thanks – kulssaka Jun 30 '16 at 10:07
  • The same elasticsearch-hadoop jar you need to add to driver class path. – Knight71 Jun 30 '16 at 10:24
  • bin/spark-submit --verbose --class "SimpleApp" --master local[4] target/simple-project-1.0.jar --jars elasticsearch-hadoop-2.3.2.jar is not working, this is the only jar required for JavaEsSpark. I also tried : ../../bin/spark-submit --verbose --class "SimpleApp" --master local[4] target/simple-project-1.0.jar --jars elasticsearch-hadoop-2.3.2.jar --driver-class-path elasticsearch-hadoop-2.3.2 --class "JavaEsSpark" sorry, i'm a beginner in this... – kulssaka Jun 30 '16 at 10:36
  • try this spark-submit --jars elasticsearch-hadoop-2.3.2.jar \ --master local[4]\ --driver-class-path elasticsearch-hadoop-2.3.2.jar \ --class "SimpleApp" target/simple-project-1.0.jar \ – Knight71 Jun 30 '16 at 10:40
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/116081/discussion-between-knight71-and-kulssaka). – Knight71 Jun 30 '16 at 10:41
  • Thanks you are right ! this command worked well: spark-submit --jars elasticsearch-hadoop-2.3.2.jar --master local[4] --driver-class-path elasticsearch-hadoop-2.3.2.jar --class "SimpleApp" target/simple-project-1.0.jar – kulssaka Jun 30 '16 at 12:24