clojure reduce function uses all cores even though it seems to be single threaded

Question

(defn sum [numbers]
  (reduce + numbers))

(def numbers (into [] (range 0 100000000)))

(time (sum numbers))

The above was the code that was ran.

Simply adding up a lot of numbers.

This line was executed in the repl multiple times:
(time (sum numbers)) Each time it almost gets all cores fully running.

Looking at jvisualvm, there were not a lot of threads created.

But this code used all 12 of the hyperthreads that were available on my 6 core laptop.

What was happening behind the scene that made this possible?

unrelated note - `(reduce + (range 0 100000000))` is significantly faster since `range` returns an object that knows how to reduce itself without creating an intermediate sequence. `(into [] ..)` forces the whole range to be realized into 1 big sequence. — xificurC, Nov 27 '20 at 16:31
Your reduce is single-threaded. The Java virtual machine may be attending to garbage collection or other tasks. The VisualVM screenshot suggests that many threads are related to RMI. — Biped Phill, Nov 27 '20 at 17:08
As mentioned in an earlier comment, there are multiple garbage collection algorithms that most JVMs can be configured to use. Some of those will take advantage of multiple parallel threads to perform garbage collection work. It seems at least possible that this is what is happening on your system. You could attempt to manually select a GC algorithm when you start the JVM that does not do this, and see if it changes how many cores are used. — andy_fingerhut, Nov 27 '20 at 20:13
This might be die to using JMX? Maybe even for jvisualvm? See https://plumbr.io/handbook/gc-tuning-in-practice/other-examples/rmi-gc — Didier A., Nov 28 '20 at 04:07

score 2 · Accepted Answer · answered Nov 28 '20 at 05:42

Thanks to the comments.

It has to do with the size of the range.

On my laptop, when it's around 70 million numbers, all is fine.

When it gets around 80 millions, the heap size grows a lot, the time taken grows very significantly, and all cores get to work. And visual vm shows more GC activity was happening.

So the comments above are probably right, it has to do with GC.

clojure reduce function uses all cores even though it seems to be single threaded

1 Answers1