Clojure error- GC overhead limit exceeded using reducer with lazy sequence

Question

; Test 1 - Using Map Reduce (Successful)
(ns example
    (:gen-class))
(require '[clojure.core.reducers :as r])

(def n 100000000000)
(time (println "map: " (reduce + 0N (map inc (range n)))))

I get: map: 5000000000050000000000N "Elapsed time: 8540888.550507 msecs"

; Test 2 - Using Map Reducer (Creates GC Error)
(ns example
     (:gen-class))
(require '[clojure.core.reducers :as r])

(def n 100000000000)
(time (println  "rmap: " (reduce + 0N (r/map inc (range n)))))

I get: Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded, compiling:... Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

; Test 3 - Using Reducer with Fold (Creates GC Error)
(ns example
  (:gen-class))
(require '[clojure.core.reducers :as r])

(def n 100000000000)
(time (println "fold: " (r/fold + (r/map inc (range n)))))

I get: Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded, compiling:... Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

Would have expected all three to produce the same result. Instead only #1 works but the other two have GC issues.

Note: You can get all three to work using smaller values of n.

I profiled this quickly in Visual VM. In the reducers cases, the heap fills up quickly with millions of [`clojure.lang.LongRange`](https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/LongRange.java) objects. From afar it looks like this is caused by the chunking code in `LongRange`, which determines and caches the ‘next’ chunk (also a `LongRange`). I have a feeling that this is something the Clojure developers themselves may want to look at. — glts, Jan 03 '17 at 19:09
There is a fix pending for this: http://dev.clojure.org/jira/browse/CLJ-1793 — Alan Thompson, Jan 04 '17 at 16:46

Clojure error- GC overhead limit exceeded using reducer with lazy sequence

0 Answers0