How to add the "--deploy-mode cluster" option to my scala code

Question

209/5000 Hello I want to add the option "--deploy-mode cluster" to my code scala:

  val sparkConf = new SparkConfig ().setMaster ("spark: //192.168.60.80:7077")

Without using the shell (the command. \ Spark-submit)

i whant to usage the " spark.submit.deployMode " in scala

score 11 · Answer 1 · answered May 12 '17 at 09:56

with SparkConfig:

//set up the spark configuration and create contexts
val sparkConf = new SparkConf().setAppName("SparkApp").setMaster("spark: //192.168.60.80:7077")

val sc = new SparkContext(sparkConf).set("spark.submit.deployMode", "cluster")

with SparkSession:

val spark = SparkSession
   .builder()
   .appName("SparkApp")
   .master("spark: //192.168.60.80:7077")
   .config("spark.submit.deployMode","cluster")
   .enableHiveSupport()
   .getOrCreate()

anshul_cached · Answer 2 · 2017-05-12T04:34:50.173

4

You can use

 val sparkConf = new SparkConf ().setMaster ("spark: //192.168.60.80:7077").set("spark.submit.deployMode","cluster")

edited May 12 '17 at 04:34

answered May 11 '17 at 12:59

anshul_cached

684
5
18

val sparkConf = new SparkConf() .set("spark.submit.deployMode" ,"cluster") .setMaster("spark://master:7077") .setAppName("wordcount") – hatem dagbouj May 11 '17 at 13:50
My address is: 192.168.60.90, is not the same in the master host: 192.168.60.80 No import, no configuration between driver and master Note that I use Eclipse maven project – hatem dagbouj May 11 '17 at 13:55
which version of spark you are using? – anshul_cached May 11 '17 at 14:36
i'm using the spark 2.1.0 – hatem dagbouj May 11 '17 at 14:50

How to add the "--deploy-mode cluster" option to my scala code

2 Answers2