Why does the Lisp reader return `(list 1 2 3)` instead of `(1 2 3)` when using reader macros?

Question

Recently, I tried to understand reader macros better. I have read an article about using reader macros to read in objects in JSON format.

With slightly adapted code from above article (it only reads (or is supposed to read) arrays [1,2,3] into lists (1 2 3))

(defun read-next-object (separator delimiter &optional (input-stream *standard-input*))
  (flet ((peek-next-char ()
           (peek-char t input-stream t nil t))
         (discard-next-char ()
           (read-char input-stream t nil t)))
    (if (and delimiter (char= (peek-next-char) delimiter))
        (progn
          (discard-next-char)
          nil)
        (let* ((object (read input-stream t nil t))
               (next-char (peek-next-char)))
          (cond
            ((char= next-char separator) (discard-next-char))
            ((and delimiter (char= next-char delimiter)) nil)
            (t (error "Unexpected next char: ~S" next-char)))
          object))))

(defun read-separator (stream char)
  (declare (ignore stream))
  (error "Separator ~S shouldn't be read alone" char))

(defun read-delimiter (stream char)
  (declare (ignore stream))
  (error "Delimiter ~S shouldn't be read alone" char))

(defun read-left-bracket (stream char)
  (declare (ignore char))
  (let ((*readtable* (copy-readtable)))
    (set-macro-character #\, 'read-separator)
    (loop
       for object = (read-next-object #\, #\] stream)
       while object
       collect object into objects
       finally (return `(list ,@objects)))))

the intent is to call read on strings and have it produce Lisp lists.

With following test code I get:

(with-input-from-string (stream "[1,2,3]")
  (let ((*readtable* (copy-readtable)))
    (set-macro-character #\[ 'read-left-bracket)
    (set-macro-character #\] 'read-delimiter)
    (read stream)))
;; => (LIST 1 2 3)

I expected to get (1 2 3) instead.

Now, when I change the current readtable "permanently", i.e. call set-macro-character in the top-level, and type [1,2,3] at the prompt, I get indeed (1 2 3).

Why, then, does

(with-input-from-string (stream "(1 2 3)")
  (read stream)))
;; => (1 2 3)

give the "expected" result? What am I missing? Is there some eval hidden, somewhere? (I'm aware of the quasi-quote above, but some in-between reasoning is missing...)

Thanks!

EDIT:

Using

(defun read-left-bracket (stream char)
  (declare (ignore char))
  (let ((*readtable* (copy-readtable)))
    (set-macro-character #\, 'read-separator)
    (loop
       for object = (read-next-object #\, #\] stream)
       while object
       collect object)))

I get what I expect. Entering '[1,2,3] at the REPL behaves like entering "real" lists.

Reading from strings also works as intended.

something like [nil,nil,nil] might be a problem... – Rainer Joswig Dec 29 '22 at 19:41 — Rainer Joswig, Dec 29 '22 at 19:41

score 1 · Accepted Answer · answered Dec 29 '22 at 18:03

1

You have

(return `(list ,@objects))

in your code. Thus [...] is read as (list ...).

Next if you use the REPL and evaluate

> [...]

Then it is as you were evaluating

> (list 1 2 3)

which returns (1 2 3).

answered Dec 29 '22 at 18:03

Rainer Joswig

136,269
10
221
346

And if I wanted to get `(1 2 3)` directly? I could use `(return objects)`. Then I'd get what I want when reading from a string. But entering `[1,2,3]` at the REPL would fail. – nosejose Dec 29 '22 at 18:08
@nosejose, you'd need to enter `'[...]` at the REPL. Alternative return a quoted list. – Rainer Joswig Dec 29 '22 at 18:33
Thanks, due to the hint it dawned on me: I can just return the loop-collection, no need for the initial quoting. – nosejose Dec 29 '22 at 18:48

Why does the Lisp reader return `(list 1 2 3)` instead of `(1 2 3)` when using reader macros?

1 Answers1