1

How can we change the parameters in Spark Submit Default Command line options in Qubole.

Though there is a option to override the values if needed under "Spark Submit Command Line Options" but this option is not available in Spark "Command Line".

Throw
  • 11
  • 2

1 Answers1

0

In case of Spark Command Line, you can pass the configuration in the code itself, as below:

/usr/lib/spark/bin/spark-submit --class org.apache.spark.examples.SparkPi -- 
master yarn-client /usr/lib/spark/spark-examples*.jar --conf 
spark.executor.instances=2 --conf spark.logConf=true

Reference Link: https://docs.qubole.com/en/latest/user-guide/analyze/compose-spark.html#compose-a-spark-application-using-the-command-line

Urvashi kohli
  • 46
  • 2