1

On my current project, I tried to deploy a 2.2 version of SPARK where on the cluster a 2.1 version is available. I looked in the SPARK documentation the way to deploy specific dependencies on a cluster which led me to use the following spark-submit:

    spark-submit --master yarn --class MainMethodSparkApp --conf spark.driver.extraClassPath=localPath-to-jar-with-ependencies
   --conf spark.executor.extraClassPath=localPath-to-jar-with-dependencies --conf spark.jars=hdfsPath-jar-with-dependencies --queue queueName --deploy-mode cluster 
   --driver-memory xx --num-executors xx --executor-memory xx --executor-core xx

The driver and executors are set using the jar with the dependencies. However, I am still getting a No.Such.Method.Exception; The SPARK documentation is a bit unclear as to which options to use to successfully deploy specific dependencies. What am I missing? Any suggestions are welcomed. Thanks a lot!

AlexCh
  • 11
  • 2

1 Answers1

0

I was finally able to make it work. Here is what I did: the necessary components are: --conf spark.driver.extraClassPath=/local-path-of-each-datanode/spark-with-depencies.jar --conf spark.executor.extraClassPath=/local-path-of-each-datanode/spark-with-depencies.jar The jar contains all the dependencies of the alternative version of Spark. Also make sure to include Yarn/Hadoop dependencies since it must must be deployed on a Yarn/Hadoop cluster

spark-submit --master yarn --class MainMethodSparkApp --conf spark.driver.extraClassPath=localPath-to-jar-with-ependencies --conf spark.executor.extraClassPath=localPath-to-jar-with-dependencies --queue queueName --deploy-mode cluster --driver-memory xx --num-executors xx --executor-memory xx --executor-core xx /local-path-on-edgeNode/smalljar.jar

The smalljar.jar is the jar launching the application and doesn't include dependencies I noticed that the fat-jar must be deployed on each and every node.

AlexCh
  • 11
  • 2