1

Spark-submit command:

[root@d03db3cedc5a opt]# bash -x $SPARK_HOME/bin/spark-submit --master spark://analytics-seed:7077 --py-files $SPARK_HOME/hello_world.py
+ '[' -z /opt/spark ']'
+ export PYTHONHASHSEED=0
+ PYTHONHASHSEED=0
+ exec /opt/spark/bin/spark-class org.apache.spark.deploy.SparkSubmit --master spark://analytics-seed:7077 --py-files /opt/spark/hello_world.py  
Exception in thread "main" java.lang.IllegalArgumentException: Missing application resource.
        at org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:241)
        at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitArgs(SparkSubmitCommandBuilder.java:160)
        at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:276)
        at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:151)
        at org.apache.spark.launcher.Main.main(Main.java:86)

When I try to run the java command directly it throws below exception:

[root@d03db3cedc5a opt]# /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el8_4.x86_64/jre/bin/java -Xmx128m -cp '/opt/spark/jars/*' org.apache.spark.launcher.Main org.apache.spark.deploy.SparkSubmit --master spark://analytics-seed:7077 --py-files /opt/spark/hello_world.py
Exception in thread "main" java.lang.IllegalStateException: Cannot find any build directories.
        at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
        at org.apache.spark.launcher.AbstractCommandBuilder.getScalaVersion(AbstractCommandBuilder.java:241)
        at org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:195)
        at org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:118)
        at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:237)
        at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:151)
        at org.apache.spark.launcher.Main.main(Main.java:86)

I have exported below ENVs:

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el8_4.x86_64/jre
export SPARK_HOME=/opt/spark
export PATH=$PATH:$SPARK_HOME/bin
export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH
export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.10.3-src.zip:$PYTHONPATH
Animesh Srivastava
  • 69
  • 5

1 Answers1

1

The command is failing due to --py-files argument.

$SPARK_HOME/bin/spark-submit \
--master spark://analytics-seed:7077 \
--py-files $SPARK_HOME/hello_world.py

After removing the argument the command should work as expected:

$SPARK_HOME/bin/spark-submit \
--master spark://analytics-seed:7077 \
hello_world.py
Animesh Srivastava
  • 69
  • 5