1

I am trying to run a Spark job using spark-submit in Windows. I am executing the below spark-submit command from command prompt.

spark-submit --driver-class-path %FILE_NAME%\config --files %FILE_NAME%\config\app.conf,%FILE_NAME%\config\log4j.properties --conf "spark.driver.extraJavaOptions=-Dcassandra.username=cassandra -Dcassandra.password=cassandra@123" --class com.sapient.main.MainApp %FILE_NAME%\lib\test.jar

Here, I am trying to pass two system properties using spark.driver.extraJavaOptions one is cassandra.username and other is cassandra.password but on executing the command, I am getting below error

The input line is too long.

However, if I pass only one system property to extraJavaOptions and without double quotes "" like below, then everything works fine.

 --conf spark.driver.extraJavaOptions=-Dcassandra.username=cassandra

But I need to pass 2 system properties. How can I resolve this? Also let me know if there is another way to pass system properties to spark which could avoid this issue.

Note: It runs in linux environment.

halfer
  • 19,824
  • 17
  • 99
  • 186
Anand
  • 20,708
  • 48
  • 131
  • 198

1 Answers1

2

You can use a conf file and pass it to spark-submit

or

pass it like below

--conf spark.driver.extraJavaOptions=-Dcassandra.username=cassandra\
--conf spark.rpc.message.maxSize=1024\
--conf spark.network,timeout=1000\
Rahul
  • 399
  • 4
  • 10
  • need to provide multiple system properties i.e. multiple key-value pairs in extraJavaOptions.. how to do that? – Anand Jan 29 '19 at 13:27
  • can you try something like --conf spark.driver.extraJavaOptions=-Dcassandra.configuration=file:/data/security.properties – Rahul Jan 29 '19 at 13:39