Not able to call "spark-submit" from within scala via system call apparently due to "--jars" parameter (having *wildcard) not being expanded

Question

Following "spark-submit" call works fine in shell

/bin/bash -c '/local/spark-2.3.1-bin-hadoop2.7/bin/spark-submit --class analytics.tiger.agents.spark.Orsp --master spark://analytics.broadinstitute.org:7077 --deploy-mode client --executor-memory 1024m --conf spark.app.id=Orsp --conf spark.executor.extraJavaOptions=-Dconfig.file=/home/unix/analytics/TigerETL3/application.conf --conf spark.driver.extraJavaOptions=-Dconfig.file=/home/unix/analytics/TigerETL3/application.conf --jars "/home/unix/analytics/TigerETL3/spark-jars/*.jar" /home/unix/analytics/TigerETL3/spark-agents.jar'

however when I merely converted it into a system call within Scala like this:

val cmd = Seq("/bin/bash", "-c", s"""/local/spark-2.3.1-bin-hadoop2.7/bin/spark-submit --class analytics.tiger.agents.spark.Orsp --master spark://analytics.broadinstitute.org:7077 --deploy-mode client --executor-memory 1024m --conf spark.app.id=Orsp --conf spark.executor.extraJavaOptions=-Dconfig.file=/home/unix/analytics/TigerETL3/application.conf --conf spark.driver.extraJavaOptions=-Dconfig.file=/home/unix/analytics/TigerETL3/application.conf --jars "/home/unix/analytics/TigerETL3/spark-jars/*.jar" /home/unix/analytics/TigerETL3/spark-agents.jar""")
import scala.sys.process._
val log = cmd.lineStream.toList
println(log.mkString)

throws an error

Warning: Local jar /home/unix/analytics/TigerETL3/spark-jars/*.jar does not exist, skipping.
Exception in thread "main" java.lang.NoClassDefFoundError: scalikejdbc/DB
        at java.lang.Class.getDeclaredMethods0(Native Method)
        at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
        at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
        at java.lang.Class.getMethod0(Class.java:3018)
        at java.lang.Class.getMethod(Class.java:1784)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:739)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: scalikejdbc.DB
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 10 more

Exception suggests that *.jars pattern is not being expanded for some reason (even though it works fine in shell). Enumerating all jars in CSV-list is not very appealing, would be a monster - 187 jars. I have tried any trick I could think of and failed miserably, haven't been so frustrated in long time.

Help appreciated ! Thanks

score 1 · Answer 1 · answered Oct 31 '18 at 10:26

1

You need to remove the double quotes "" while specifying --jars. Can you try this?

val cmd = Seq("/bin/bash", "-c", s"""/local/spark-2.3.1-bin-hadoop2.7/bin/spark-submit --class analytics.tiger.agents.spark.Orsp --master spark://analytics.broadinstitute.org:7077 --deploy-mode client --executor-memory 1024m --conf spark.app.id=Orsp --conf spark.executor.extraJavaOptions=-Dconfig.file=/home/unix/analytics/TigerETL3/application.conf --conf spark.driver.extraJavaOptions=-Dconfig.file=/home/unix/analytics/TigerETL3/application.conf --jars /home/unix/analytics/TigerETL3/spark-jars/*.jar /home/unix/analytics/TigerETL3/spark-agents.jar""")
import scala.sys.process._
val log = cmd.lineStream.toList
println(log.mkString)

answered Oct 31 '18 at 10:26

Paul

1,176
3
12
27

Removed the double quotes as suggested and now I get a `java.lang.ClassNotFoundException: analytics.tiger.agents.spark.Orsp` which makes no sense since I know for sure that class is located in the jar provided as most-right parameter in the call. – Nasko Oct 31 '18 at 14:30
I read somewhere that bash does not expand the '*' when in script-mode (whereas in interactive mode it does and all jars in that folder are properly added) – Nasko Oct 31 '18 at 14:50

score 1 · Answer 2 · answered Oct 31 '18 at 18:05

Ok, I figured it out. I had to read through Spark's scripts to realize that if SPARK_HOME and JAVA_HOME are missing Spark will go through bunch of steps trying to infer it. My initial Scala command (including the double quotes) was perfectly fine - I only had to define those 2 variables like this

val cmd = Seq("/bin/bash", "-c", s"""JAVA_HOME=/broad/software/free/Linux/redhat_7_x86_64/pkgs/jdk1.8.0_121 SPARK_HOME=/local/spark-2.3.1-bin-hadoop2.7 /local/spark-2.3.1-bin-hadoop2.7/bin/spark-submit --class analytics.tiger.agents.spark.Orsp --master spark://analytics.broadinstitute.org:7077 --deploy-mode client --executor-memory 1024m --conf spark.app.id=Orsp --conf spark.executor.extraJavaOptions=-Dconfig.file=/home/unix/analytics/TigerETL3/application.conf --conf spark.driver.extraJavaOptions=-Dconfig.file=/home/unix/analytics/TigerETL3/application.conf --jars "/home/unix/analytics/TigerETL3/spark-jars/*.jar" /home/unix/analytics/TigerETL3/spark-agents.jar""")

and it's working like a charm.

Not able to call "spark-submit" from within scala via system call apparently due to "--jars" parameter (having *wildcard) not being expanded

2 Answers2