3

I am trying to utilise all the resources which I have on the cluster to run the Spark job. I have Cloudera Manager installed on all of the nodes. This is the command which I use to submit the job.

spark-submit --master yarn 
             --deploy-mode cluster 
             file:///[spark python file]
             file://[app argument 1]
             file://[app argument 2]

During the execution I receive following error:

diagnostics: Application application_1450777964379_0027 failed 2 times due to AM Container for appattempt_1450777964379_0027_000002 exited with  exitCode: 1

Any ideas how to fix it will be much appreciated.

EDIT 1 The machine where Spark is installed is not accessible by WEB UI I tried to download the sources and read little bit more about the exception.

------------------------------------------------------------
| Job | Description                                        | 
------------------------------------------------------------
| 0   | saveAsTextFile at NativeMethodAccessorImpl.java:-2 | 
------------------------------------------------------------
mr. Holiday
  • 1,780
  • 2
  • 19
  • 37
  • Do you have permissions to save the file in your filesystem? – BlueMoon93 Jan 13 '16 at 10:39
  • As it is a cluster, on each machine I have created the supergroup and each user in that group has read/write permission to the folder where the output should be written – mr. Holiday Jan 13 '16 at 11:11

1 Answers1

0

Taken from here,

If the path starts with file:// or hdfs://, the path becomes ile:// or dfs://. If the path is absolute the first slash is removed.

There is no particular reason for it and needs to be fixed.

Try using an absolute path instead of file://

Community
  • 1
  • 1
BlueMoon93
  • 2,910
  • 22
  • 39
  • I saw this thread, but I got absolutely same results nothing new, the error description is the same – mr. Holiday Jan 13 '16 at 11:52
  • 1
    I would advise you to use the Spark mailing list then. They can probably provide more help than SO community (I doubt this thread will have other answers unless someone puts a bounty on it) http://apache-spark-user-list.1001560.n3.nabble.com/ – BlueMoon93 Jan 13 '16 at 11:56