Spark Streaming gets stopped without errors after ~1 minute

Question

When I make spark-submit for the Spart Streaming job, I can see that it's running during approximatelly 1 minute, and then it's stopped with the final status SUCCEEDED:

16/11/16 18:58:16 INFO yarn.Client: Application report for application_XXXX_XXX (state: RUNNING)
16/11/16 18:58:17 INFO yarn.Client: Application report for application_XXXX_XXX (state: FINISHED)

I don't understand why it gets stopped, while I expect it to run for an undefined time and be triggered by messages received from the Kafka queue. In logs I can see all the println outputs, and there are no errors.

This is a short extract from the code:

val conf = new SparkConf().setAppName("MYTEST")
val sc = new SparkContext(conf)
sc.setCheckpointDir("~/checkpointDir")

val ssc = new StreamingContext(sc, Seconds(batch_interval_seconds))

val rootLogger = Logger.getRootLogger()
rootLogger.setLevel(Level.ERROR)

println("Dividing the topic into partitions.")
val inputKafkaTopicMap = inputKafkaTopic.split(",").map((_, kafkaNumThreads)).toMap
val messages = KafkaUtils.createStream(ssc, zkQuorum, group, inputKafkaTopicMap).map(_._2)

messages.foreachRDD(msg => {
  msg.foreach(s => {
    if (s != null) {
      //val result = ... processing goes here
      //println(result)
    }
  })
})

// Start the streaming context in the background.
ssc.start()

This is my spark-submit command:

/usr/bin/spark-submit --master yarn --deploy-mode cluster --driver-memory 10g --executor-memory 10g --num-executors 2 --conf "spark.executor.extraJavaOptions=-XX:+UseG1GC \
-XX:+AlwaysPreTouch" --class org.test.StreamingRunner test.jar param1 param2

When I open Resource Manager, I see that no job is RUNNING and the spark streaming job gets marked as FINISHED.

I would try commenting out `rootLogger.setLevel(Level.ERROR)` to get more verbose output. Likely whatever is killing your job isn't tagged `ERROR` so it's being filtered out of the logs. Also, it looks like you are missing your call to `ssc.awaitTermination` at the end of your code. — Eric M., Nov 16 '16 at 18:10
@EricM.: Ok, thanks. Let me test it without this line of code. I will tell you the result in a couple of minutes. — duckertito, Nov 16 '16 at 18:11
@EricM.: I tried to run it with `ssc.awaitTermination`, but there was the same problem. But anyway, let me check it as well again. — duckertito, Nov 16 '16 at 18:17
@EricM.: You are right. The problem was because of the lack of `ssc.awaitTermination`. I somehow mixed it with `ssc.awaitTerminationOrTimeout`. Thanks. — duckertito, Nov 16 '16 at 18:24
@EricM.: I appreciate if you can publish the answer so that I could accept it. By the way do you know how enable Spark Streaming print-outs in the terminal, so that instead of `Application report for application_XXXX_XXX...` I could see the messages of code? — duckertito, Nov 16 '16 at 18:47

score 1 · Accepted Answer · answered Nov 16 '16 at 19:19

Your code is missing a call to ssc.awaitTermination to block the driver thread.

Unfortunately there's no easy way to see the printouts from inside your map function on the console, since those function calls are happening inside YARN executors. Cloudera Manager provides a decent look at the logs though, and if you really need them collected on the driver you can write to a location in HDFS and then scrape the various logs from there yourself. If the information that you want to track is purely numeric you might also consider using an Accumulator.

Spark Streaming gets stopped without errors after ~1 minute

1 Answers1