Scala slow builds: development approaches to avoid

Question

First of all, incremental builds via SBT are pretty awesome, generally in the < 1sec range. However, sometimes you have to do a full clean/compile, or, in the case of incremental builds, you make a change to one file which then triggers the compilation of dozens of other files.

This is when Scala development becomes less...fun, as the resulting slowdown in work flow can encourage context switching (check email, latest Stackoverflow threads, etc.), which subtly makes one less productive

So, what are the development approaches to avoid in order to improve full clean/compile builds, and (ideally), change-one-file-without-recompiling-half-the-application incremental builds?

Examples I can think of:
1) is it better to have a thousand+ line do-it-all scala file, or several files split up?
2) can I have my cake (pattern) or will that inflate build times?
3) can I have pimp'd x,y,z library pattern, or better to find another way?
4) are package objects (with implicits) a build time killer?
5) nested objects and traits?
6) implicit methods/parameters or stop being clever and be explicit?

Concretely, I'm thinking of ditching a cake pattern DAO I came up with and consolidating into ScalaQuery case class + companion object + minimal database provider trait. That alone will shed 20 scala files.

The application is small enough (120 scala + 10 java files) to refactor now without too much hassle. Obviously as a scala application grows, so too will the build times, just based on LOCs alone. I'm just trying to see where to trim the fat and where not to bother (i.e. keep things as they are) so current and future applications benefit from the expressiveness that scala affords without needlessly inflating build times.

Thanks for some examples of your experience of the good, the bad, and the ugly of scala development vis a vis build times.

Actually, I wonder why you need the `clean` so often that it annoys you? Is something being messed up? In my experience, `clean` is very rarely needed. One of the cases is if you work with a snapshot dependency and you need to update that. In those cases I found `rm -r lib_managed/jars` and a subsequent compile faster. — 0__, Jul 21 '12 at 11:43
It's true, clean is not often required, but the need does arise (corrupt/missing class file and package objects, the primary culprit for me) and of course deployment requires a full clean/compile, which can ne a hassle when, ooops, missed that typo, have to clean/compile again. This doesn't even cover incremental builds where a single code change can, instead of recompiling the changed file, cascade into tens of files (self types, cake pattern, etc. come into play here), which in my small application turns < 1 second into > 10 seconds, a huge difference. — virtualeyes, Jul 21 '12 at 13:07

score 3 · Answer 1 · answered Jul 22 '12 at 16:59

I've noticed that type members can force rebuilds in places you would not expect. For example:

foo.scala:

object foo {
    class A {
        type F = Float
    }
    def z: Int = 8
}

bar.scala:

object bar {
    def run { println(foo.z) }
}

Changing the value of z does not force bar to be recompiled. Changing the type of F does, even though bar never refers to F or even to A. Why, I have no idea (Scala 2.9.1).

score 2 · Accepted Answer · answered Jul 30 '12 at 17:54

2

Have a look at how incremental recompilation works in SBT.

It's roughly this:

Find all classes whose publicly visible API have changed
Invalidate all of its dependents, its dependents' dependents, and so on.

For the purposes of SBT, a "dependent" is both a user of the class and a class defined in the same file.

Owen's example for foo.scala could even be this, and you'd see the issue:

object foo {
  def z: Int = 8
}

object foo2 {
  class A { ... }
}

Good practices:

Separate files for separate classes
Fine-grained interfaces
Use the same level of abstraction in companion objects as in their companion classes; if the companion object reaches up through layers of abstraction, pull it into a separate class and file.

answered Jul 30 '12 at 17:54

cldellow

359
6
14

+1, thanks, oddly enough, I refactored over the weekend and wound up putting, for example model case class + companion mapper object + DAO trait and implementation class all in ONE file. Why did you not reply sooner? ;-) I need to fire up the pre-refactored project to see the incremental build time diffs, but I am seeing a lot of: (compiling 71 Scala and 8 Java files) on a single change to Foo model/dao class file, which is to say, slooooooow. This may have been the case pre-refactor but need to verify.. – virtualeyes Jul 30 '12 at 18:23
set logLevel in Global := Level.Debug is your friend - it shows the precise files invalidated at each phase of the invalidation fan out. If you see a high-level file invalidating a bunch of low-level files, dig in, it probably can be reworked to avoid that. – cldellow Jul 30 '12 at 19:29
thanks, doesn't provide any clues, however. Getting, "Sources indirectly invalidated by:" and then empty Set() references; as to why the compiler grinds away on 68 scala and 4 java files on Foo DAO change during incremental builds, I don't know, but would really like to see what's happening under the hood (i.e. exactly which files are being compiled) – virtualeyes Jul 30 '12 at 21:03
Are you saying you change the one scala/java source file for the DAO and it shows 1 iteration with 79 directly invalidated files? Sounds suspicious. Can you post a gist of the sbt output of a compile with debug loglevel? – cldellow Jul 30 '12 at 21:16
I'm saying that sbt shows zero invalidated files with debug flag set. "Sources indirectly invalidated by:" and then empty Set() references, means, apparently, there are no invalidated files. The compiler then proceeds to recompile 2/3 of my application, listing the number of scala and java files, but not showing me any details about which files are being compiled – virtualeyes Jul 30 '12 at 22:26
here it is, the reality check: `set scalacOptions in Compile += "-print"` that is one ginormous amount of code generation occuring under the hood ;-) Unfortunate that I now see a single DAO file change cascading into a related controller, and then on and on from there, literally 2/3 of my application recompiled, yikes. Will have to see how I can decouple things (if even possible) – virtualeyes Jul 31 '12 at 00:37

Scala slow builds: development approaches to avoid

2 Answers2

Linked