Why does this blow up the heap in Lispworks?

Question

I'm trying to solve Problem 14 in Project Euler (find the longest Collatz sequence between 1 and 1000000).

My code consists of a recursive, memoized function to calculate the length of Collatz sequences followed by a loop to find the maximum. Please see the code below:

(defparameter *collatz* (make-hash-table))
(setf (gethash 1 *collatz*) 0)

(defun n-collatz (n)
  (if (gethash n *collatz*)
      (gethash n *collatz*)
      (setf (gethash n *collatz*)
            (if (evenp n)
                (1+ (n-collatz (/ n 2)))
                (1+ (n-collatz (1+ (* n 3))))))))

(loop for i from 1 to 1000000 maximize (n-collatz i))

This worked fine in Clozure, but caused a heap overflow in Lispworks. As it exited ungracefully, I couldn't find out what happened. Actually, I don't understand why this consumes so much heap space—the biggest recursion sequence is 300-something calls. Did I miss some inefficiency in the code?

Any input is appreciated. Further comments on the code are appreciated as well.

PS: I'm using Lispworks Personal Edition, which imposes a limit on heap size.

UPDATE I did try compiling as Rainer Joswig suggested, but it didn't help.

Regarding coredump's and sds's comments, or is indeed better than if in this case, but I can't substitute the hash table for a vector because the collatz sequence goes up about 50% of the time. After running the code, the hash table has some 2.5 million entries.

Finally, and strangely, I managed to reproduce the bug while testing the syntax of a longish loop (a million iterations) that just juggled some variables and didn't collect anything at all. I lost the code unfortunately—LispWorks just went poof, alas. My best guess is that there's some leak or other memory management glitch in LispWorks' entrails.

I cannot reproduce the overflow (no Lispworks here); all I can say is that `(if X X Y)` is also written as `(or X Y)` — coredump, Oct 08 '15 at 15:49
Are you using LispWorks Personal Edition? If so, then remember that hash tables are usually implemented upon arrays with lots of unused slots, so your hash table might grow enough to make LispWorks Personal Edition blow up on its heap limit, followed by its signature prompt exit without a chance to recover. — acelent, Oct 08 '15 at 16:01
UPDATE: I did try compiling as Rainer Joswig suggested, but it didn't help. I'm wondering if there's any way of making Lispworks compile stuff more aggressively. — Paulo Mendes, Dec 09 '15 at 19:25

score 4 · Answer 1 · answered Oct 08 '15 at 17:23

One thing would be to make sure that n-collatz is compiled:

(compile 'n-collatz)

Or use the compiler via the usual IDE commands.

Code entered into the LispWorks Listener is interpreted otherwise:

CL-USER 140 > (defun n-collatz (n)
                (if (gethash n *collatz*) (gethash n *collatz*)
                  (setf (gethash n *collatz*)
                        (if (evenp n)
                            (1+ (n-collatz (/ n 2)))
                          (1+ (n-collatz (1+ (* n 3))))))))
N-COLLATZ

CL-USER 141 > #'n-collatz
#<interpreted function N-COLLATZ 40600020CC>

CL-USER 142 > (compile 'n-collatz)
N-COLLATZ
NIL
NIL

CL-USER 143 > #'n-collatz
#<Function N-COLLATZ 4060007054>

score 4 · Answer 2 · answered Feb 28 '17 at 13:14

LispWorks just went poof, alas. My best guess is that there's some leak or other memory management glitch in LispWorks' entrails.

You are using the personal edition of LW, which has a memory limit, and this stuff reaches this limit. It raised a dialog that says that, and then quits.

The commercial editions of LW run it without any problem.

score 3 · Accepted Answer · answered Oct 08 '15 at 16:12

I see two inefficiencies here:

You are using a hash table indexed by a contiguous integer sequence. You should probably be using an (extensible) vector instead.
Your recursion is not tail-recursion; you might prefer to optimize that.

Admittedly, none of these could explain a heap exhaustion.

Leo · Answer 4 · 2015-10-09T23:35:29.073

I think there might be a problem with having to resize the hash table each time you call

(setf (gethash n collatz))

with a number n higher than the current table size. When you call make-hash-table without a size parameter, the system chooses an implementation dependent value. Every time that this value is exceeded, the table has to be resized during execution time, which consumes a lot of resources and might lead to the crash you mention. To solve this issue, you can create the table with a value that you know will not be exceeded:

(defparameter *collatz* (make-hash-table :size 1000000)).

score 1 · Answer 5 · answered May 02 '19 at 12:18

I'm running LispWorks 7.1.1 64bit on a Mac, using the interpreter

CL-USER 1 > (defparameter *collatz* (make-hash-table))
*COLLATZ*

CL-USER 2 > (setf (gethash 1 *collatz*) 0)
0

CL-USER 3 > (defun n-collatz (n)
              (if (gethash n *collatz*)
                  (gethash n *collatz*)
                (setf (gethash n *collatz*)
                      (if (evenp n)
                          (1+ (n-collatz (/ n 2)))
                        (1+ (n-collatz (1+ (* n 3))))))))
N-COLLATZ

CL-USER 4 > (loop for i from 1 to 1000000 maximize (n-collatz i))

Stack overflow (stack size 15998).
  1 (continue) Extend stack by 50%.
  2 (abort) Return to top loop level 0.

Type :b for backtrace or :c <option number> to proceed.
Type :bug-form "<subject>" for a bug report template or :? for other options.

So above shows a 'stack overflow', not a 'heap overflow'. Notice that one can resize the stack and continue.

Now I tried it again in a fresh LispWorks, but compiling the function:

CL-USER 1 > (defparameter *collatz* (make-hash-table))
*COLLATZ*

CL-USER 2 > (setf (gethash 1 *collatz*) 0)
0

CL-USER 3 > (defun n-collatz (n)
              (if (gethash n *collatz*)
                  (gethash n *collatz*)
                (setf (gethash n *collatz*)
                      (if (evenp n)
                          (1+ (n-collatz (/ n 2)))
                        (1+ (n-collatz (1+ (* n 3))))))))
N-COLLATZ

CL-USER 4 > (compile 'n-collatz)
N-COLLATZ
NIL
NIL

CL-USER 5 > (loop for i from 1 to 1000000 maximize (n-collatz i))
524

The compiled code works fine without the need to grow the stack.

Why does this blow up the heap in Lispworks?

5 Answers5