0

I submit a spark app to mesos cluster(running in cluster mode), and pass java system property through "--drive-java-options=-Dkey=value -Dkey=value", however these system properties are not available at runtime, seems they are not set. --conf "spark.driver.extraJavaOptions=-Dkey=value" doesn't work either

More details: the command is 

bin/spark-submit --master mesos://10.3.101.119:7077 --deploy-mode cluster --class ${classname} --driver-java-options "-Dconfiguration.http=http://10.3.101.119:9090/application.conf" --conf "spark.executor.extraJavaOptions=-Dconfiguration.http=http://10.3.101.119:9090/application.conf" ${jar file}

I have a two-node mesos cluster, one node both runs master and slave, and the other runs slave only. I submit the spark application on master node.

Internally, the application hopes to read a configuration file from java system property "configuration.http", if the property is not available, the application will load a default file from the root of the classpath. When I submit the application, from the logs, i saw the default configuration file is loaded.

And the actual command to run the application is 

"sh -c '/home/ubuntu/spark-1.6.0/bin/spark-submit --name ${appName} --master mesos://zk://10.3.101.184:2181/mesos/grant --driver-cores 1.0 --driver-memory 1024M --class ${classname} ./${jar file} '"

from here you can see the system property is lost

Tobi
  • 31,405
  • 8
  • 58
  • 90
Grant
  • 500
  • 1
  • 5
  • 18

1 Answers1

1

You might have a look at this blog post which recommends using an external properties file for this purpose:

$ vi app.properties
spark.driver.extraJavaOptions   -Dconfiguration.http=http://10.3.101.119:9090/application.conf
spark.executor.extraJavaOptions –Dconfiguration.http=http://10.3.101.119:9090/application.conf

Then try to run this via

bin/spark-submit --master mesos://10.3.101.119:7077 --deploy-mode cluster --class ${classname} —-properties-file app.properties ${jar file}

See

Community
  • 1
  • 1
Tobi
  • 31,405
  • 8
  • 58
  • 90
  • Do I have to put that file to all of the nodes? We don't run HDFS, so I have to put copies of the properties file at a special path on all nodes? – Grant Mar 09 '16 at 16:04
  • My understanding is that the file only has to be present on the node where you execute `spark-submit`, but I haven't tried it... – Tobi Mar 09 '16 at 16:35