I am using livy to start a spark job inside aws EMR
and see sometimes it keeps failing as below.
The following is from the livy Log
23/08/15 01:27:09 ERROR SparkYarnApp: Error whiling refreshing YARN state
java.lang.IllegalStateException: No YARN application is found with tag livy-batch-0-qejnaq0g in 120 seconds.
This may be because 1) spark-submit fail to submit application to YARN; or
2) YARN cluster doesn't have enough resources to start the application in time. Please check Livy log and YARN log to know the details.
at org.apache.livy.utils.SparkYarnApp.org$apache$livy$utils$SparkYarnApp$$getAppIdFromTag(SparkYarnApp.scala:183)
at org.apache.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:242)
at org.apache.livy.utils.SparkYarnApp$$anonfun$1$$anonfun$4.apply(SparkYarnApp.scala:239)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.livy.utils.SparkYarnApp$$anonfun$1.apply$mcV$sp(SparkYarnApp.scala:239)
at org.apache.livy.Utils$$anon$1.run(Utils.scala:97)
Exception in thread "Thread-15" java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:72)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at org.apache.livy.utils.LineBufferedStream$$anon$1.run(LineBufferedStream.scala:40)
23/08/15 01:27:10 WARN BatchSession$: spark-submit exited with code 143
23/08/15 01:37:49 INFO BatchSessionManager: Deleting BatchSession 0 because it was inactive for more than 3600000.0 ms.
23/08/15 01:37:49 INFO BatchSession: Stopping BatchSession 0...
23/08/15 01:37:49 INFO BatchSession: Stopped BatchSession 0.
not sure what to do
the below are the applications installed in aws EMR
