-1

I'm trying to switch cluster manager from standalone to 'YARN' in Apache Spark that I've installed for learning.

I read following thread to understand which cluster type should be chosen

However, I'd like to know the steps/syntax to change the cluster type. Ex: from Standalone to YARN or from YARN to Standalone.

Ash
  • 1,180
  • 3
  • 22
  • 36
  • What's about your hardware/software? YARN is independent from Spark resource manager, suitable for many others applications. As I know, YARN is not provided with Spark, so you must ensure, YARN is installed. – Dmitry R. Bushkov Jan 10 '18 at 08:28

2 Answers2

2

In spark there is one function name as --master that can helps you to execute your script on yarn Cluster mode or standalone mode.

Run the application on local mode or standalone used this with spark-submit command 

--master Local[*]

or

  --master spark://192.168.10.01:7077 \
  --deploy-mode cluster \

Run on a YARN cluster

 --master yarn 
 --deploy-mode cluster

For more information kindly visit this link. https://spark.apache.org/docs/latest/submitting-applications.html

Sahil Desai
  • 3,418
  • 4
  • 20
  • 41
2

If you are not running through command line then you can directly set this master on SparkConf object.

sparkConf.setMaster(http://path/to/master/url:port) in cluster mode

or

sparkConf.setMaster(local[*]) in client/local mode
TheCodeCache
  • 820
  • 1
  • 7
  • 27