Reducing memory usage in a simple Clojure program

Question

I'm trying to solve the Fibonacci problem on codeeval. At first I wrote it in the usual recursive way and, although I got the right output, I failed the test since it used ~70MB of memory and the usage limit is 20MB. I found an approximation formula and rewrote it to use that thinking it was the heavy stack usage that was causing me to exceed the limit. However, there doesn't seem to be any reduction.

(ns fibonacci.core
  (:gen-class))

(use 'clojure.java.io)

(def phi (/ (+ 1 (Math/sqrt 5)) 2))

(defn parse-int 
  "Find the first integer in the string"
  [int-str]
  (Integer. (re-find  #"\d+" int-str)))

(defn readfile
  "Read in a file of integers separated by newlines and return them as a list"
  [filename]
  (with-open [rdr (reader filename)]
    (reverse (map parse-int (into '() (line-seq rdr))))))

(defn fibonacci
  "Calculate the Fibonacci number using an approximatation of Binet's Formula. From
  http://www.maths.surrey.ac.uk/hosted-sites/R.Knott/Fibonacci/fibFormula.html"
  [term]
  (Math/round (/ (Math/pow phi term) (Math/sqrt 5))))

(defn -main
  [& args]
  (let [filename (first args)
        terms (readfile filename)]
      (loop [terms terms]
        ((comp println fibonacci) (first terms))
        (if (not (empty? (rest terms)))
          (recur (rest terms))))))

(-main (first *command-line-args*))

Sample input:

Sample output:

0
1
1
2
3
5
8
13
21
34
55
89
144
233
12586269025

Their input is clearly much larger than this and I don't get to see it. How can I modify this code to use dramatically less memory?

Edit: It's impossible. Codeeval knows about the problem and is working on it. See here.

I doubt stack usage was your problem - the problem with the recursive Fibonacci implementation is the breadth of the call tree, not the depth. — Alex, Jun 17 '14 at 16:28
But is not many shallow stacks still characterized as stack usage? — Alex Holt, Jun 17 '14 at 16:50
Each thread in the JVM has a single call stack, the size of which changes over time as method calls are pushed onto or pulled off of the stack. Using the recursive Fibonacci implementation, the call stack for computing `F(n)` is never deeper than `n` at any one point in time. — Alex, Jun 17 '14 at 17:28
For one, you don't need to have the whole file in memory at once. Open the file, iterate over each line and close it. I doubt it, but perhaps one of the test cases is a file with ten million ones. — Diego Basch, Jun 17 '14 at 17:45
Does codeeval work? https://www.codeeval.com/leaderboard/?type=user&filtered=yes&page=1&months_back=&country=&city=&languages=9&save_filter=no shows no clojure solutions. — KobbyPemson, Jun 17 '14 at 18:13
@KobbyPemson, It's very strange that page is empty and makes me wonder if it's even possible to solve their problems in Clojure. — Alex Holt, Jun 18 '14 at 03:26
I think [the link you posted in your edit to the OP](https://getsatisfaction.com/codeeval/topics/clojure_time_memory_stats_seem_wrong) explains it. Apparently CodeEval has been "working on it" for almost a year now... :/ — Dave Yarwood, Jun 18 '14 at 12:57

score 4 · Accepted Answer · answered Jun 18 '14 at 04:07

4

Codeeval is broken for Clojure. There are no accepted Clojure solutions listed and there is a message from the company two months ago saying they are still working on the problem.

Darn.

answered Jun 18 '14 at 04:07

Alex Holt

165
8

Randy Hudson · Answer 2 · 2014-06-17T18:07:22.337

3

Presumably the problem is that the input file is very large, and your readfile function is creating the entire list of lines in memory.

The way to avoid that is to process a single line at a time, and not hold on to the whole sequence, something like:

(defn -main [& args]
  (let [filename (first args)]
    (with-open [rdr (reader filename)]
       (doseq [line (line-seq rdr)]
         (-> line parse-int fibonacci println)))))

edited Jun 17 '14 at 18:07

answered Jun 17 '14 at 17:54

Randy Hudson

494
3
4

There's no substantial difference but the file is definitely the problem. Can you think of anything else? – Alex Holt Jun 18 '14 at 03:24

score -1 · Answer 3 · edited May 23 '17 at 11:57

-1

Are you getting a stack overflow error? If so, Clojure supports arbitrary precision, so it might help to try using /' instead of /. The apostrophe versions of +, -, * and / are designed to coerce numbers into BigInts to prevent stack overflow.

See this question.

edited May 23 '17 at 11:57

Community

1
1

answered Jun 17 '14 at 16:52

Dave Yarwood

2,866
1
17
29

1

Stack overflow and integer overflow are completely different things. – Alex Jun 17 '14 at 17:14
Ah, I understand now, after reading your comment above about a recursive Fibonacci implementation's call stack for calculating `F(n)` never being deeper than `n`. I figured the problem might be that Clojure needed to start using bigints after a certain point, but it turns out that 12586269025 + 233 is way smaller than that threshold... we'd still have to go up about 10 levels of magnitude to reach bigint territory. Never mind. – Dave Yarwood Jun 17 '14 at 19:16

Reducing memory usage in a simple Clojure program

3 Answers3