2

in my pyspark program i have as, 

from pyspark import SparkConf, SparkContext, SQLContex  
conf=SparkConf()  
conf.setAppName("spark_name")  
conf.set("spark.dynamicAllocation.enabled", "true")  
conf.set("spark.shuffle.service.enabled", "true")  
sc = SparkContext(conf=conf)

And running my pyspark program as 

./spark-submit --master yarn-cluster PySparkProgramPath

then this job failed with 

Exception in thread "main" org.apache.spark.SparkException: Application application_1482268372614_0318 finished with failed status

after i run yarn logs -applicationId application_1482268372614_0318
it showing the error 

ERROR ApplicationMaster: User class threw exception: java.io.IOException: Cannot run program
ERROR ApplicationMaster: SparkContext did not initialize after waiting for 100000 ms. Please check earlier log output for errors.

[EDIT] Not facing this error when i submit this in yarn-client mode

user491
  • 175
  • 1
  • 4
  • 20
  • I believe it should be --master yarn, not yarn-cluster, see http://spark.apache.org/docs/latest/running-on-yarn.html It may be trying to access master on 'yarn-cluster' host – Denis Makarenko Feb 06 '17 at 21:10
  • @DenisMakarenko, `--master yarn-cluster` means master is yarn and deploy mode is cluster. Getting the same error even if i mention both separately. – user491 Feb 06 '17 at 21:16

0 Answers0