reading large command output with clojure

Question

I'm using the sh function from the clojure.java.shell command to read the very large output of a command. The output is around 60meg of data.

I keep getting java.lang.OutOfMemoryError. Is there a way to open a sort of pipe that would let me read the output and parse it into a vector . Like a lazy sequence for the output of the command?

Basically the data is a large array of byte that I want to convert to just numbers and put into a vector.

score 4 · Accepted Answer · answered Nov 01 '10 at 19:29

clojure.java.shell/sh will always return a non-lazy string

A solution (doesn't handle closing, environment passing and encoding) using lazy line-seq on an BufferedReader:

(->> (.exec (Runtime/getRuntime) "YOUR_LONG_RUNNING_COMMAND ARG ...")
    .getInputStream
    clojure.java.io/reader
    line-seq
    (map YOUR-FUNCTION))

reading large command output with clojure

1 Answers1