0

have a file of s-expressions, which includes foreign language characters, that I am reading in as follows:

(defun test (file)
  (with-open-file (stream file)
    (loop while (read stream nil nil))))

It reads the file without errors in ccl 1.8, but throws an error under 1.9:

? (test "/users/markklein/desktop/naples.text")
> Error: Reader error: Illegal symbol syntax.
> While executing: CCL::%PARSE-TOKEN, in process Listener(5).
> Type cmd-. to abort, cmd-\ for a list of available restarts.
> Type :? for other options.
1 > 

Does anyone have any ideas what is going wrong, and how to fix it? I can send the data file on request.

Rainer Joswig
  • 136,269
  • 10
  • 221
  • 346
  • 3
    "I can send the data file on request." Please include it in the question. But narrow down the problem just to the text that's causing the problem. E.g., cut out half the file; if the problem is still there, then it's in the remaining half, otherwise put the half back and take out the other half. You should be able to get it down to the problematic s-expression. It's important that you provide a complete example, meaning enough so that *we* can reproduce the problem, and a *minimal* example, meaning that there's as little "extra" as possible, since it just gets in the way. – Joshua Taylor Jun 17 '14 at 02:32
  • 3
    Plus, often times the process of trying to create a minimal complete example is enough to show you where the problem is. – Joshua Taylor Jun 17 '14 at 02:33
  • 2
    http://article.gmane.org/gmane.lisp.openmcl.devel/9193 ? – Ben Hyde Jun 17 '14 at 14:06
  • @BenHyde, I think it should be somehow posted as an answer. – Mark Karpov Jun 17 '14 at 14:22
  • Usually the `file-position` will be recoverable information when you have a `reader-error` on a `file-stream`. I would try the `:?` option and see if anything looks promising. If you get the `file-position` you can open the file in your favorite editor and jump to that character. Failing that, I would locate the failing s-exp by wrapping the `read` call in a `print` call. – m-n Jun 17 '14 at 21:41

1 Answers1

2

Ben Hyde noted in a comment that R. Matthew Emerson spoke to this issue on the Clozure mailing list, pointing out that Clozure CL's default external format has been changed to :utf-8. As a result, he suggested this alternative:

We changed the default external format to :utf-8 in the 1.9 release. This is probably tripping you up. Try specifying an appropriate external-format explicitly, e.g.,

(defun test (file)
  (with-open-file (stream file :external-format :iso-8859-1)
    (loop while (read stream nil nil))))
Community
  • 1
  • 1
Joshua Taylor
  • 84,998
  • 9
  • 154
  • 353