8

I am trying to understand what the different metrics that Spark Streaming outputs mean and I am slightly confused what is the difference between the Processing Time, Total Delay and Processing Delay of the last batch ?

I have looked at the Spark Streaming guide which mentions the Processing Time as a key metric for figuring if the system is falling behind, but other places such as "Pro Spark Streaming: The Zen of Real-Time Analytics Using Apache Spark" speak about using Total Delay and Processing Delay. I have failed to find any documentation that lists all the metrics produced by Spark Streaming with explanation what each one of them means.

I would appreciate if someone can outline what each of these three metrics means or point me to any resources that can help me understand that.

Yuval Itzchakov
  • 146,575
  • 32
  • 257
  • 321
Zak
  • 85
  • 1
  • 5

3 Answers3

19

Let's break down each metric. For that, let's define a basic streaming application which reads a batch at a given 4 second interval from some arbitrary source, and computes the classic word count:

inputDStream.flatMap(line => line.split(" "))
            .map(word => (word, 1))
            .reduceByKey(_ + _)
            .saveAsTextFile("hdfs://...")

  • Processing Time: The time it takes to compute a given batch for all its jobs, end to end. In our case this means a single job which starts at flatMap and ends at saveAsTextFile, and assumes as a prerequisite that the job has been submitted.

  • Scheduling Delay: The time taken by Spark Streaming scheduler to submit the jobs of the batch. How is this computed? As we've said, our batch reads from the source every 4 seconds. Now let's assume that a given batch took 8 seconds to compute. This means that we're now 8 - 4 = 4 seconds behind, thus making the scheduling delay 4 seconds long.

  • Total Delay: This is Scheduling Delay + Processing Time. Following the same example, if we're 4 seconds behind, meaning our scheduling delay is 4 seconds, and the next batch took another 8 seconds to compute, this means that the total delay is now 8 + 4 = 12 seconds long.

A live example from a working Streaming application:

Streaming application

We see that:

  • The bottom job took 11 seconds to process. So now the next batches scheduling delay is 11 - 4 = 7 seconds.
  • If we look at the second row from the bottom, we see that scheduling delay + processing time = total delay, in that case (rounding 0.9 to 1) 7 + 1 = 8.
Community
  • 1
  • 1
Yuval Itzchakov
  • 146,575
  • 32
  • 257
  • 321
  • 1
    Thanks for the exhaustive answer, exactly what I was looking for. – Zak Nov 02 '16 at 18:05
  • @Yuval Is there a way that I can get the metrics like , total memory used by app till now , no. of containers used etc, so that I can compare spark streaming with other realtime processing tools – JSR29 Jun 27 '17 at 08:31
  • 1
    @JSR29 There is a REST API launched by the driver, [usually at port 4040](https://spark.apache.org/docs/latest/monitoring.html), which does expose quite a lot of information about the running app. – Yuval Itzchakov Jun 27 '17 at 08:43
0

We're experiencing stable processing time, however increasing scheduling delay.

Based on the answer, the scheduling delay should be influenced only by processing time of previous runs.

Spark is running only streaming, nothing else.

Time window is 1 minute, processing 120K records.

enter image description here

Tomas Bartalos
  • 1,256
  • 12
  • 29
0

If your window is 1 minute, and the average processing time is 1 minute 7 seconds, you have a problem : each batch will delay the next one by 7 seconds.

Your processing time graph shows a stable processing time, but always higher than batch time.

I think after a given amount of time, your driver will crash on GC overhead limit exceeded, as it will be full of pending batch waiting to be excecuted.

You can change this by reducing the processing time so that it goes under the expected microbatch max duration (requires code and/or resources allocation changes), or increase the microbatch size, or go to continuous streaming.

Rgds

Th.
  • 21
  • 2