0

When i use the below setting to start the spark application (default is yarn-client mode) works fine

spark_memory_setting="--master yarn --conf spark.dynamicAllocation.enabled=true --conf spark.shuffle.service.enabled=true --conf spark.yarn.queue=ciqhigh --conf spark.dynamicAllocation.initialExecutors=50 --conf spark.dynamicAllocation.maxExecutors=50 --executor-memory 2G --driver-memory 4G"

ISSUE

Whereas when i change the deploy mode as cluster,application not starting up. Not even throwing any error to move on.

 spark_memory_setting="--master yarn-cluster --deploy-mode=cluster --conf spark.dynamicAllocation.enabled=true --conf spark.shuffle.service.enabled=true --conf spark.yarn.queue=ciqhigh --conf spark.dynamicAllocation.initialExecutors=50 --conf spark.dynamicAllocation.maxExecutors=50 --executor-memory 2G --driver-memory 4G"

LOG

18/01/08 01:21:00 WARN Client: spark.yarn.am.extraJavaOptions will not take effect in cluster mode

This is the last line from the logger.

Any suggestions most welcome.

One important think to highlight here, the spark application which am trying to deploy starts the apache thrift server. After my searching, i think its coz of thrift am not able to able to run yarn in cluster mode. Any help to run in cluster mode.

Murali
  • 13
  • 1
  • 6

1 Answers1

0

the option --master yarn-cluster is wrong.. this is not a valid master url it should be just "yarn" instead of "yarn-cluster".. just cross check..

TheCodeCache
  • 820
  • 1
  • 7
  • 27
  • I tried with yarn alone as you said, still no luck.My suspect now is, are we able to run apache thrift in cluster mode? !!! – Murali Jan 10 '18 at 09:53