0

My problem is I am trying to execute a fatjar assembly generated with sbt, but I am not able to make it works. When I run my code from the IDE (intelliJ) it works perfectly. However, once I generate the assembly with sbt, I am getting 2 different errors depending on the command I use to execute the jar: Executing the jar from java jvm:

java -jar data-self-service-assembly-0.1.jar

After doing some stuff, I get this error on log:

[Ljava.lang.StackTraceElement;@7435a578.
No FileSystem for scheme: hdfs

Executing the jar from spark-submit:

spark2-submit --master yarn --name self-service data-self-service-assembly-0.1.jar

After doing some stuff, I get this error on log:

[Ljava.lang.StackTraceElement;@290c266c.
No suitable driver

I am running out on ideas, so any help that I can get to resolve my issue will be very appreciated.

Thank you.

James
  • 2,954
  • 2
  • 12
  • 25

2 Answers2

0

Taking a stab at it

1 . Check if you have below snippet in your code 

.setMaster("local[2]")

You can remove that

2 . Check the path of spark config . 

spark-submit --verbose --master yarn-cluster  --files   --properties-file {spark_conf_file_location} --class {main_class}  {jar_location}
Sandeep Das
  • 1,010
  • 9
  • 22
  • Thanks @Sandeep Das for your comment. I have tried with both of your recommendations and it doesn't work yet:_( – James Jun 06 '18 at 14:27
0

finally I have been able to resolve the issue this way:

  • I accessed the spark history server using this url: http://quickstart.cloudera:18089/
  • I get the log of the execution to get more information about it
  • I saw the classpath used by spark was this one: Classpath","/opt/cloudera/parcels/SPARK2-2.2.0.cloudera2-1.cdh5.12.0.p0.232957/lib/spark2/jars/
  • So I copied the jar files for jdbc in that path
  • Then I submitted the job with this command: spark2-submit --master yarn --name self-service data-self-service-assembly-0.1.jar

The problem is that I didn't expect to have to make these actions, because I thought sbt would be able to assembly all the needed jars and recognize where they are later, but here there is some conflict between the classpath used by spark2-submit and sbt assembly.

Hope it helps.

James
  • 2,954
  • 2
  • 12
  • 25