Tuning clojure reducers library performance

Question

Why is mapping/reducing with reducers library has worse performance than normal map/reduce?

user=> (time (reduce + (map inc (range 100000000))))
"Elapsed time: 14412.627966 msecs"
5000000050000000

user=> (time (reduce + (r/map inc (range 100000000))))
... (C-c)

user=> (time (r/reduce + (r/map inc (range 100000000))))
....(C-c)

I had two kill the later two as it takes indefinitely long. What is wrong here?

EDIT: It seems like other languages also have similar problems. Scala seems to be breaking at a million only. Why do Scala parallel collections sometimes cause an OutOfMemoryError? . While clojure reducers are faster than normal at a million.

@JiacaiLiu clojure.core.reducers, http://clojure.org/reducers — FUD, Feb 26 '14 at 06:36

score 6 · Accepted Answer · answered Feb 26 '14 at 09:44

To complement @a-webb's answer, this is a compiler bug and an involved one to really fix. (See this post for more details.)

Another way to work around the problem is to use a fuse:

(defn once-seq
  "Returns a sequence on which seq can be called only once."
  [coll]
  (let [a (atom coll)]
    (reify clojure.lang.Seqable
      (seq [_]
        (let [coll @a]
          (reset! a nil)
          (seq coll))))))

and then:

=> (time (r/reduce + (r/map inc (once-seq (range 100000000)))))
"Elapsed time: 3692.765 msecs"
5000000050000000

score 2 · Answer 2 · edited May 23 '17 at 12:10

Performance is grinding to a halt as memory is being exhausted. If you keep waiting, you would most likely encounter a memory error. Creating a reducer holds the head of the collection in a closure. Thus, the huge lazy-sequence ties up memory as it becomes realized.

Here's what's happening, distilled

user=> (def n 100000000)
#'user/n

user=> (time (dorun (range n)))
"Elapsed time: 12349.034702 msecs"

Now the same, but from within a closure

user=> (defn foo [x] (fn [f] (f x)))
#'user/foo

user=> (time ((foo (range n)) dorun))
OutOfMemoryError GC overhead limit exceeded ... (sometime later)

Compare to

(time (do (doall (range n)) nil))
OutOfMemoryError GC overhead limit exceeded ... (sometime later)

The suspect closure in reducers

user=> (source r/folder)
(defn folder
  "Given a foldable collection, [...]"
  {:added "1.5"}
  ([coll xf]
     (reify
      clojure.core.protocols/CollReduce
      (coll-reduce [_ f1]
                   (clojure.core.protocols/coll-reduce coll (xf f1) (f1)))
   ...

Christophe Grand has a nice post on how to compose reducers in a lazy fashion.

So, I totally missed that two posts later in the same blog Christophe Grand covers this exact problem. I defer to [cgrand's answer](http://stackoverflow.com/a/22037468/1756702) with linked post. — A. Webb, Feb 26 '14 at 18:25

score 1 · Answer 3 · answered Feb 26 '14 at 06:57

Reducers do not really work well with lazy lists while normal reduce does.

To get the real benefit out of reducers you need a non-lazy collection, e.g. a vector and you need to use fold instead of reduce.

 (def v (into [] (range 10000000)))
 (time (reduce + (map inc v)))
 ;; "Elapsed time: 896.79 msecs"
 (time (reduce + (r/map inc v)))
 ;; "Elapsed time: 671.947 msecs" 
 (time (r/fold + (r/map inc v)))
 ;; "Elapsed time: 229.45 msecs"

Reducers work with the fork/join Framework which requires large chunks of data. In a lazy (chunked) sequence you do not have these large chunks, so fork/join cannot work well.

There is a talk by Rich Hickey on reducers that explain the concepts really well: https://vimeo.com/45561411

Fork/join is not applicable to r/map or r/reduce here. Also, r/reduce and normal reduce are the same. — A. Webb, Feb 26 '14 at 07:00
Add another `0` to your range as in the question and you may also hit a memory error on the first line depending on your JVM settings. — A. Webb, Feb 26 '14 at 07:06

Tuning clojure reducers library performance

3 Answers3