12

I followed this page and ran the SparkPi example application on YARN in yarn-cluster mode.

http://spark.apache.org/docs/latest/running-on-yarn.html

I don't see the output of the program at the end (which is the result of the computation in this case). When I run it in the yarn-client mode (--master yarn-client), I see an output like this:

Pi is roughly 3.138796

Where does the standard out go in the yarn-cluster mode?

Muffintop
  • 543
  • 4
  • 11

1 Answers1

11

After much poking around, I found this in the spark-0.9.0 doc.

Examine the output (replace $YARN_APP_ID in the following with the "application identifier" output by the previous command) (Note: YARN_APP_LOGS_DIR is usually /tmp/logs or $HADOOP_HOME/logs/userlogs depending on the Hadoop version.)

$ cat $YARN_APP_LOGS_DIR/$YARN_APP_ID/container*_000001/stdout

Pi is roughly 3.13794

I wish they put this instruction in the 1.1.0 documentation too.

Community
  • 1
  • 1
Muffintop
  • 543
  • 4
  • 11
  • The logs information is in the Debugging Your Application section of the documentation.It also describes how you can use the 'yarn logs' command to view the logs. – Tom Panning Mar 19 '15 at 14:35