2

Im doing a performance comparison between executing multiple subsequent transformations on collections in Scala that are strict (eagerly performed evaluation), and non-strict (lazily performed evaluation).

def time[R](block: => R): R = {
    val t0 = System.nanoTime()
    val result = block // call-by-name
    val t1 = System.nanoTime()
    println("Elapsed time: " + (t1 - t0)/1e9 + "s")
    result
}

/* A view on a collection makes all transformations lazy, which makes it
   possible to combine multiple transformations into one. */

// The non-lazy (eager) version:
time {
    (1 to 1e7.toInt).map(_ + 2).map(x => {
        if(x % 2 != 0) -x else x
    }).sum
}

// The lazy version using a view:
time {
    (1 to 1e7.toInt).view.map(_ + 2).map(x => {
        if(x % 2 != 0) -x else x
    }).force.sum
}

On my laptop, the first run of the eager version is slower than the lazy version. See timings below.

Eager version: 2.4 s

Lazy verion: 0.7 s

However, starting from the second run, both of them take about 0.7 seconds. Why?

Runtime environment:

  • Scala 2.11.7
  • Java 1.8
  • OS X 10.10
Community
  • 1
  • 1
qed
  • 22,298
  • 21
  • 125
  • 196

2 Answers2

3

Skip the cold run

When profiling code, the first run/iteration which is called the cold run is usually not measured as it will incur some initial costs for the OS and/or the runtime environment.

In the case of Java and Scala code executed on the JVM, the first execution of certain instructions may require the loading and parsing of classes and methods into memory, thus creating some overhead.

The JVM will also replace, e.g., some math operations with native instructions during the first iteration, causing a speedup of execution for subsequent iterations.

Felix Glas
  • 15,065
  • 7
  • 53
  • 82
  • How do you then explain the lazy version of the same function performs as well in the cold run as in the "warm run"? – qed Oct 17 '15 at 14:01
  • @qed The initial overhead will occur on the first use of any dependent classes or methods, and it has nothing to do with the syntactic construct of a loop iteration. The initial cost is probably applied during the first execution of the eager version, while the first execution of the _lazy_ version employs the loaded methods in memory and is thus faster. – Felix Glas Oct 18 '15 at 22:58
1

Getting accurate benchmarks is notoriously difficult - if your code above reflects the actual code you are using to get these results, you are unlikely to getting reliable data.

You should use a real benchmarking tool such as https://github.com/ktoso/sbt-jmh.

Once you're sure that there is a significant difference in performance, then you can start investigating how/why.

melps
  • 1,247
  • 8
  • 13