Unexpected behavior with `eval-when`

Question

The following fragment of CL code does not work as I expected with CCL running SLIME. If I first compile and load the file using C-c C-k, and then run

(rdirichlet #(1.0 2.0 3.0) 1.0)

in the SLIME/CCL REPL, I get the error

value 1.0 is not of the expected type DOUBLE-FLOAT.
   [Condition of type TYPE-ERROR]

It works with SBCL. I expected the (setf *read-default-float-format* 'double-float)) to allow me to use values like 1.0. If I load this file into CCL using LOAD at the REPL it works. What am I missing?

(eval-when (:compile-toplevel :load-toplevel :execute)
  (require :asdf) (require :cl-rmath) (setf *read-default-float-format* 'double-float))

(defun rdirichlet (alpha rownum)
  ;;(declare (fixnum rownum))
  (let* ((alphalen (length alpha))
    (dirichlet (make-array alphalen :element-type '(double-float 0.0 *) :adjustable nil :fill-pointer nil :displaced-to nil)))
    (dotimes (i alphalen)
      (setf (elt dirichlet i) (cl-rmath::rgamma (elt alpha i) rownum)))
    ;; Divide dirichlet vector by its sum
    (map 'vector #'(lambda (x) (/ x (reduce #'+ dirichlet))) dirichlet)))

UPDATE: I forgot to mention my platform and versions. I'm using Debian squeeze x86. The version of SLIME is from Debian unstable, 1:20120525-2. CCL is the 1.8 release. I tried it both with the upstream binaries from http://svn.clozure.com/publicsvn/openmcl/release/1.8/linuxx86/ccl, and a binary package created by me - see Package ccl at mentors.debian.net. The result was the same in each case.

It seems probable that this issue is SLIME specific. It would be helpful if people could comment whether they see this behavior or not. Also, what is the equivalent of C-c C-k in SLIME, if one is running CCL in basic command line mode? (LOAD filename), or something else? Or, to ask a slightly different question, what CCL function is C-c C-k calling?

I notice that calling C-c C-k on the following code

(eval-when (:compile-toplevel :load-toplevel :execute)
      (require :asdf) (require :cl-rmath) (setf *read-default-float-format* 'double-float))

(print *read-default-float-format*)

produces DOUBLE-FLOAT, though even though *read-default-float-format* at the REPL immediately afterwards gives SINGLE-FLOAT.

UPDATE 2: It looks like, as Rainer said, that the compilation occurs in a separate thread.

Per the function all-processes in Threads Dictionary

printing all-processes from the buffer using C-c C-k gives

(#<PROCESS worker(188) [Active] #x18BF99CE> #<PROCESS repl-thread(12) [Semaphore timed wait] #x187A186E> #<PROCESS auto-flush-thread(11) [Sleep] #x187A1C9E> #<PROCESS swank-indentation-cache-thread(6) [Semaphore timed wait] #x186C128E> #<PROCESS reader-thread(5) [Active] #x186C164E> #<PROCESS control-thread(4) [Semaphore timed wait] #x186BE3BE> #<PROCESS Swank Sentinel(2) [Semaphore timed wait] #x186BD0D6> #<TTY-LISTENER listener(1) [Active] #x183577B6> #<PROCESS Initial(0) [Sleep] #x1805FCCE>)
CL-USER> (all-processes)

and in the REPL gives

(#<PROCESS repl-thread(12) [Active] #x187A186E> #<PROCESS auto-flush-thread(11) [Sleep] #x187A1C9E> #<PROCESS swank-indentation-cache-thread(6) [Semaphore timed wait] #x186C128E> #<PROCESS reader-thread(5) [Active] #x186C164E> #<PROCESS control-thread(4) [Semaphore timed wait] #x186BE3BE> #<PROCESS Swank Sentinel(2) [Semaphore timed wait] #x186BD0D6> #<TTY-LISTENER listener(1) [Active] #x183577B6> #<PROCESS Initial(0) [Sleep] #x1805FCCE>)

so it seems #<PROCESS worker(188) [Active] #x18BF99CE> is the thread that is doing the compilation. Of course, there still remains the question of why these variables are local to a thread, and also why SBCL does not behave the same way.

Answer 1

I can see that with CCL and some older (which I use) SLIME, too. Haven't tried it with a newer SLIME.

It does not happen with SBCL or LispWorks.

*read-default-float-format* is one of the I/O variables of Common Lisp. Something like WITH-STANDARD-IO-SYNTAX binds them to the standard values and, on exit, restores the previous values. So I suspect that CCL's SLIME code has such a binding in effect. This is confirmed by setting other I/O variables like *read-base* - they are also bound.

CCL, Emacs and SLIME has some layers of code which makes it slightly complex to debug this.

• Emacs runs the ELISP code of SLIME and talks to SWANK.
• SWANK is the backend of SLIME and is Common Lisp code running in CCL.
• SWANK has portable code and some CCL-specific code.

On the Emacs side the SLIME/ELISP function SLIME-COMPILE-AND-LOAD-FILE is used.

On the SWANK side the Common Lisp function swank:compile-file-for-emacs gets called. Later SWANK:LOAD-FILE gets called.

I don't see where the I/O variables are bound - maybe it is in the CCL networking code?

This seems to be the answer:

If CCL has thread-local I/O variables, then the compilation happens in another thread and won't change the I/O bindings in the REPL thread and also not in any other thread.

This is a difference between various CL implementations. Since the CL standard does not specify threads or processes, it is also not specified if special bindings are global or have a thread-local binding by default...

If you think about it, protecting a thread's I/O variables against changes from other threads makes sense...

So, the correct way to deal with it should be to make sure in each thread independently that the right I/O variable values are in effect.

---

Let me expand a bit why things are like they are.

Typically a Common Lisp implementation can run more than one thread. Some implementations also allow concurrent threads running at the same time on different cores. These threads can do very different things: one could run a REPL, another one could answer an HTTP request, one could load data from the disk and another one could read the contents of an Email. Lisp in this case runs several different tasks inside one Lisp system.

Lisp has several variables which determine the behavior of I/O operations. For example which format floats are when read or which base integer numbers are in when read. The letter is done by `read-base.

Now imagine that the above disk reading thread has set its *read-base* to 16 for some purpose. Now you change the global in another thread to 8 and then suddenly all other threads have base 8. The consequence: the disk reading thread will suddenly see *read-base* 8 instead of 16 and work differently.

So it makes sense to prevent this in some way. The simplest is that in each thread the running code has their own bindings for the I/O values and then changing the *read-base* won't have effects on other threads. These bindings are usually introduced by LET or a function call. Typically the code would be responsible to bind the variables.

Another way to prevent it is to give each thread a number of initial bindings, which should for example include the I/O bindings. CCL does that. LispWorks for example does that, too. But not for the I/O variables.

Now each Lisp might give you a non-portable way to change the thread-local top binding (CCL has that, too - for example (setf ccl:symbol-value-in-process) ). Still it would not mean that it might change the binding in effect in the REPL. Since the REPL itself is a piece of Lisp code, running in a thread and it could have been setting up its own bindings.

In CCL you can also set the global static binding: (CCL::%SET-SYM-GLOBAL-VALUE sym value). But if you use such functionality you are probably doing something wrong or you have a good reason.

Some background for CCL: http://clozure.com/pipermail/openmcl-devel/2011-June/012882.html

Lore

• Don't try to change global bindings from one thread visible for other threads.
• Shield your code from changes in other threads by binding the crucial variables to the correct values.
• Write a function which sets up the variable values to some values in a controllable fashion.