Performance of line counting with scalaz-stream

Question

I've translated the imperative line counting code (see linesGt1) from the beginning of chapter 15 of Functional Programming in Scala to a solution that uses scalaz-stream (see linesGt2). The performance of linesGt2 however is not that great. The imperative code is about 30 times faster than my scalaz-stream solution. So I guess I'm doing something fundamentally wrong. How can the performance of the scalaz-stream code be improved?

Here is my complete test code:

import scalaz.concurrent.Task
import scalaz.stream._

object Test06 {

val minLines = 400000

def linesGt1(filename: String): Boolean = {
  val src = scala.io.Source.fromFile(filename)
  try {
    var count = 0
    val lines: Iterator[String] = src.getLines
    while (count <= minLines && lines.hasNext) {
      lines.next
      count += 1
    }
    count > minLines
  }
  finally src.close
}

def linesGt2(filename: String): Boolean =
  scalaz.stream.io.linesR(filename)
    .drop(minLines)
    .once
    .as(true)
    .runLastOr(false)
    .run

def time[R](block: => R): R = {
  val t0 = System.nanoTime()
  val result = block
  val t1 = System.nanoTime()
  println("Elapsed time: " + (t1 - t0) / 1e9 + "s")
  result
}

time(linesGt1("/home/frank/test.txt"))        //> Elapsed time: 0.153122057s
                                              //| res0: Boolean = true
time(linesGt2("/home/frank/test.txt"))        //> Elapsed time: 4.738644606s
                                              //| res1: Boolean = true
}

I'm no `scalaz-stream` expert, but this looks reasonable to me (although I'd probably use something like `.once.runLast.run.nonEmpty`). There's a lot of overhead involved in using `drop` to step through the lines one-by-one, and I'd guess that's what you're seeing here. — Travis Brown, Sep 17 '13 at 18:03

score 2 · Accepted Answer · answered Sep 18 '13 at 14:26

When you are doing profiling or timing, you can use Process.range to generate your inputs to isolate your actual computation from the I/O. Adapting your example:

time { Process.range(0,100000).drop(40000).once.as(true).runLastOr(false).run }

When I first ran this, it took about 2.2 seconds on my machine, which seems consistent with what you were seeing. After a couple runs, probably after JIT'ing, I was consistently getting around .64 seconds, and in principle, I don't see any reason why it couldn't be just as fast even with I/O (see discussion below).

In my informal testing, the overhead per 'step' of scalaz-stream seems to be about 1-2 microseconds (for instance, try Process.range(0,10000). If you have a pipeline with multiple stages, then each step of the overall stream will consist of several other steps. The way to think about minimizing the overhead of scalaz-stream is just to make sure that you're doing enough work at each step to dwarf any overhead added by scalaz-stream itself. This post has more details on this approach. The line counting example is kind of a worst case, since you are doing almost no work per step and are just counting the steps.

So I would try writing a version of linesR that reads multiple lines per step, and also make sure you do your measurements after JIT'ing.

Performance of line counting with scalaz-stream

1 Answers1