can scala compiler utilize multi-core CPU?

Question

Is there any benefit from compiling Scala code on multi-CPU machine?

(Asking about scala compilation, not about scala code for multi-core processing)

score 4 · Answer 1 · answered Aug 21 '19 at 14:59

Triplequote Hydra is a commercial multicore Scala compiler enabling parallel compilation of Scala code. Speedups are in the order of 2x (50% faster) with a quad-core machine, and can scale up well on hardware with more cores.

Note that the Scala -Ybackend-parallelism <n> flag only enables parallelization of the last phase of the compiler pipeline, which is the bytecode emission. Bytecode emission usually takes very little time compared to other compiler phases (e.g., typechecking), hence the gains are small or negligible depending on the code.

(Full disclosure, I'm a founder of Triplequote.)

score 3 · Answer 2 · answered Mar 27 '19 at 17:01

3

You might want to check some experimental compiler flags. Scala 2.12.5 introduced -Ybackend-parallelism N, which let you emit bytecode in parallel - PR claimed 4-5% improvement with 8 backend threads.

https://kubuszok.com/2018/speed-up-things-in-scalac-and-sbt/

answered Mar 27 '19 at 17:01

skjagini

3,142
5
34
63

Thanks. So it's still experimental? The current one still uses only 1 CPU? – orshachar Mar 28 '19 at 15:08

score 0 · Answer 3 · answered May 27 '19 at 18:16

When I build a large Scala project my CPU is maxed on all virtual cores (quad core with hyperthreading). There is nothing special like Hydra at play, just a bunch of modules to build, which seem to go in parallel. Don’t forget that GC needs its own CPU time also.

One good way to experiment would be to spin up a cloud instance with a ton of cores, clone & build your project, and time it. When it next comes time to upgrade my PC I will certainly be doing this to see how many cores are worth having.

can scala compiler utilize multi-core CPU?

3 Answers3