Clojure: Creating a lazy sequence of byte chunks from an input stream

Question

Given an input stream I would like to create a lazy sequence of the data in the form of byte-arrays (chunks). Here's my try:

(defn- read-or-nil [stream]
  (let [buf (byte-array 2)]
    (when-not (= -1 (.read stream buf))
      buf)))

(defn byte-chunk-seq [stream]
  (cons (read-or-nil stream) (lazy-seq (byte-chunk-seq stream))))

(with-open [rdr (java.io.FileInputStream. "/tmp/wut")]                                                                                                                                                                                         
  (take 2 (byte-chunk-seq rdr)))

In the last statement, where I'm testing the code, I get a:

IOException Stream Closed java.io.FileInputStream.readBytes (FileInputStream.java:-2).

If I change the statement to be a take 1 then it returns fine, but that doesn't help me much. Does anyone have any ideas why this wouldn't work?

Jeremy · Accepted Answer · 2013-06-29T04:19:38.797

There are a couple issues.

First, your lazy sequence isn't entirely correct. The whole body of the function should by wrapped in lazy-seq, and it should pass in either the cons to continue the sequence, or nil to terminate it.

(defn byte-chunk-seq [stream]
  (lazy-seq (if-let [buf (read-or-nil stream)]
              (cons buf (byte-chunk-seq stream))
              nil)))

Second, take is also lazy. So the lazy sequence is not going to be realized until after with-open has closed the stream. You can avoid this error by wrapping a doall around the lazy sequence to realize it before the stream is closed.

(with-open [rdr (java.io.FileInputStream. "/tmp/wut")]
  (doall (take 2 (byte-chunk-seq rdr))))

Clojure: Creating a lazy sequence of byte chunks from an input stream

1 Answers1