do not nest defun (use flet or labels), sbcl REPL does not report all conditions (warnings)

Question

Tests performed on SBCL 1.3.1

I define function xx inside of function xxx. (Originally xx was recursive and there were invariants used from the xxx closure. However the recursion is not important here, xx just returns t) xxx only calls xx. Hence xxx is expected to return t also.

xxx is called twice inside of the function call-xxx. For this issue to manifest (that of the condition) it must be called twice. Though no state is shared between the calls, so why this would matter is curious.

call-xxx is passed as #'call-xxx to apply, inside of a handler-case. It takes no arguments, so it is just applied to nil. handler-case pulls it out and says it throws a condition.

When xx is instead defined outside of xxx, there is no condition, and the expected result of t is returned. This is shown in the second part of the code shown below, where we have ww, www, call-www, etc. Also when handler-case is not used, there is no exception reported by the REPL.

[This example is stubbed out from a testing framework, thus when call-xxx is said to throw an exception the test fails. Yet, when the test runs manually (see run-call-xx) it passes without an exception, thus creating a contradiction and making it difficult to debug the apparently failed test.]

What causes this condition? Should the condition handler call be different? Is this an SBCL bug?

Here is the code:

(defun test (test-function)
  (handler-case (apply test-function '()) (condition () ':exception))
  ;;;  (apply test-function '()) ;; returns t, no exception
  )

;;---------------------------------------------------------------
;; throws exception, but shouldn't (?)
;;
  (defun xxx ()
    (defun xx () t) ; note comments, should be a flet or labels form
    (xx))

  (defun call-xxx ()
    (xxx)  ;; #'xxx must be called twice for the exception to appear
    (xxx)
    t)

  ;; call-xxx throws exception when run from test, but shouldn't
  (defun run-test-call-xxx ()
    (test #'call-xxx))

  ;; no problem here, call-xxx returns t when called directly
  (defun run-call-xxx ()
    (call-xxx))

;;--------------------------------------------------------
;; works fine  
;;  pulled out the nested definition of #'ww from #'www
;;
  (defun ww () t)

  (defun www ()
    (ww))

  (defun call-www ()
    (www) 
    (www)
    t)

  (defun run-test-call-www ()
    (test #'call-www))

And here it is running:

§sbcl> sbcl
This is SBCL 1.3.1.debian, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.
* (load "src/test-xxx")

T
* (run-test-call-xxx)

:EXCEPTION
* (run-test-call-www)

T
* 

added here is the test function redefined to do the reply without the handler. I've done this so as to see what the REPL reports.

(defun test (test-function)

  ;;; -> this does not get handled
  ;;; (handler-case (apply test-function '()) (serious-condition () ':exception))

  ;;; -> this triggers the handler
  ;;; (handler-case (apply test-function '()) (condition () ':exception))

  (apply test-function '()))

And this is what happens in the REPL:

§sbcl> sbcl
This is SBCL 1.3.1.debian, an implementation of ANSI Common Lisp....
distribution for more information.
* (load "src/test-handler.lisp")

T
* (run-test-call-xxx)

T
* 

As you can see the REPL does not print a warning.

Note comments, defun is only top level, so one handle recursion within a closure using flet. With defun the internal function will be defined upon each entry, though a warning is issued, that isn't making it to the REPL (though handler case does see it, that is what is going on here)

(defun f (s too-big-for-stack-invariant)
    (defun r (s) ;; note comments, should be a labels form, not defun
        ;; changes s and checks for termination
        ;; makes use of the very large invariant data
        ...
        r(s))
   ;; some stuff making use of r(s)
   )

Answer 1

Two problems with the code:

1. defun in Common Lisp defines top-level definition. Even if it appears inside another function (this way it's different from Scheme's define). Thus, function xx is redefined every time xxx is called.
2. Condition condition will catch even something "not worth mentioning" (see CLHS on handler-case). Since redefinition of xx produces a warning, it will catch it. If you don't do the handler-case warning will show in REPL but the correct result will be produced.

Answer 2

I think here the problem is that you are intercepting any condition no matter what. SBCL in this cases signals a condition about xx being redefined with defun (not an error, but a warning as it's possibly unintended).

Changing the code to:

(defun test (test-function)
  (handler-case (apply test-function '())
    (condition (x)
      (print x)
      ':exception)))

will show #<SB-KERNEL:REDEFINITION-WITH-DEFUN {1002BE3E93}>

The logical problem is that you should not declare to be able to handle conditions you don't know: you should handle the conditions you know and let others to be handled by who knows. A condition may be resolved by the proper handler by just letting the program to continue running (as it would be the case here).

This is the major difference between conditions and exceptions: unwinding is performed only later if the handler decides it's the right thing to do. With say C++ instead once an exception is thrown all is already lost at lower call stack levels when the exception is handled.