I have installed and deployed giraph-1.4.0 using the Hadoop yarn profile with the following command
mvn -Phadoop_yarn -Dhadoop.version=2.7.0 -DskipTests package
I tried running the SimpleShortestPathsComputation example
$HADOOP_HOME/bin/hadoop jar $GIRAPH_HOME/giraph-examples/target/giraph-examples-1.4.0-SNAPSHOT-for-hadoop-2.7.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/hduser/input/tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/hduser/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false,giraph.logLevel=debug
My application completed with status
22/12/11 19:27:26 INFO yarn.GiraphYarnClient: Completed Giraph: org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total running time: 10 minutes, 27 seconds.
The error in the logs
22/12/11 19:05:54 INFO master.BspServiceMaster: setJobState: {"_applicationAttemptKey":-1,"_stateKey":"FAILED","_superstepKey":-1}
22/12/11 19:05:54 FATAL master.BspServiceMaster: failJob: Killing job giraph_yarn_application_1670740884304_0004
22/12/11 19:05:54 FATAL master.BspServiceMaster: failJob: exception java.lang.IllegalStateException: ******* WORKERS [Worker(hostname=lap13162 hostOrIp=lap13162, MRtaskID=1, port=30001)] FAILED *******
22/12/11 19:05:54 ERROR master.MasterThread: masterThread: Master algorithm failed with RuntimeException
java.lang.RuntimeException: BspServiceMaster (YARN profile) is FAILING this task, throwing exception to end job run.
at org.apache.giraph.master.BspServiceMaster.failJob(BspServiceMaster.java:376)
at org.apache.giraph.master.BspServiceMaster.setJobStateFailed(BspServiceMaster.java:323)
at org.apache.giraph.master.BspServiceMaster.barrierOnWorkerList(BspServiceMaster.java:1427)
at org.apache.giraph.master.BspServiceMaster.coordinateSuperstep(BspServiceMaster.java:1661)
at org.apache.giraph.master.MasterThread.run(MasterThread.java:131)
Caused by: java.lang.IllegalStateException: ******* WORKERS [Worker(hostname=lap13162 hostOrIp=lap13162, MRtaskID=1, port=30001)] FAILED *******
... 4 more
22/12/11 19:05:54 FATAL graph.GraphTaskManager: uncaughtException: OverrideExceptionHandler on thread org.apache.giraph.master.MasterThread, msg = java.lang.RuntimeException: BspServiceMaster (YARN profile) is FAILING this task, throwing exception to end job run., exiting...
java.lang.IllegalStateException: java.lang.RuntimeException: BspServiceMaster (YARN profile) is FAILING this task, throwing exception to end job run.
at org.apache.giraph.master.MasterThread.run(MasterThread.java:204)
Caused by: java.lang.RuntimeException: BspServiceMaster (YARN profile) is FAILING this task, throwing exception to end job run.
at org.apache.giraph.master.BspServiceMaster.failJob(BspServiceMaster.java:376)
at org.apache.giraph.master.BspServiceMaster.setJobStateFailed(BspServiceMaster.java:323)
at org.apache.giraph.master.BspServiceMaster.barrierOnWorkerList(BspServiceMaster.java:1427)
at org.apache.giraph.master.BspServiceMaster.coordinateSuperstep(BspServiceMaster.java:1661)
at org.apache.giraph.master.MasterThread.run(MasterThread.java:131)
Caused by: java.lang.IllegalStateException: ******* WORKERS [Worker(hostname=lap13162 hostOrIp=lap13162, MRtaskID=1, port=30001)] FAILED *******
... 4 more
22/12/11 19:05:54 WARN server.NIOServerCnxn: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.base/java.lang.Thread.run(Thread.java:829)
Container id: container_1670740884304_0004_01_000003
Exit code: 2
Stack trace: ExitCodeException exitCode=2:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Any ideas as to why this has failed would be greatly appreciated!
