Clojure way of reading large files and transforming data therein

Question

I am processing a Subrip subtitles file which is quite large and need to process it one subtitle at a time. In Java, to extract the subtitles from file, I would write a method with following signature:

Iterator<Subtitle> fromSubrip(final Iterator<String> lines);

The use of Iterator gives me two benefits:

The file is never in the memory in its entirety, nor is any of its transformed stage.
An abstraction wherein I can loop over a collection of Subtitle objects without the memory overhead.

Since iterators are by nature imperative and mutable, they're probably not idiomatic in Clojure. So what is the Clojure way to deal with this sort of situation?

@Ankur, the difference between streams (i.e. lazy sequences) and iterators is, as I understand from my Scala experience, that streams are purely functional and all of stream will stay in memory until the last reference to it is relinquished and GC picks it up. This is not the case with iterators which are mutable and can immediately allow the memory occupied by the previously traversed part to be reclaimed by the GC. — missingfaktor, Apr 14 '13 at 09:35
@Ankur, I don't and would like to know if Clojure's implementation of lazy sequence behaves differently. — missingfaktor, Apr 14 '13 at 09:35
Check out : http://stackoverflow.com/questions/3247045/how-are-lazy-sequences-implemented-in-clojure — Ankur, Apr 14 '13 at 10:11

score 3 · Answer 1 · edited May 23 '17 at 12:16

3

As Vladimir said, you need to handle the laziness and file closing correctly. Here's how I did it, as shown in "Read a very large text file into a list in clojure":

(defn lazy-file-lines 
  "open a (probably large) file and make it a available as a lazy seq of lines"
  [filename]
  (letfn [(helper [rdr]
                  (lazy-seq
                    (if-let [line (.readLine rdr)]
                      (cons line (helper rdr))
                      (do (.close rdr) nil))))]
         (helper (clojure.java.io/reader filename))))

edited May 23 '17 at 12:16

Community

1
1

answered Apr 14 '13 at 15:30

JohnJ

4,753
2
28
40

I can almost see how this doesn't consume stack... but not quite. The first call to helper (if the file isn't empty) returns a lazy-seq of a cons of a line and a nested call of helper... I guess I see it --- the outer call _returns_ and the inner call doesn't get called till needed. Sound right? If this weren't lazy it would stackoverflow, I think. Nice solution. – Reb.Cabin Oct 11 '16 at 01:46

chen_767 · Answer 2 · 2014-11-21T08:21:40.383

read all files from a directory, a lazy way.

using go black and channel.

code:

(ns user
  (:require [clojure.core.async :as async :refer :all 
:exclude [map into reduce merge partition partition-by take]]))

(defn read-dir [dir]
  (let [directory (clojure.java.io/file dir)
        files (filter #(.isFile %) (file-seq directory))
        ch (chan)]
    (go
      (doseq [file files]
        (with-open [rdr (clojure.java.io/reader file)]
          (doseq [line (line-seq rdr)]
            (>! ch line))))
      (close! ch))
    ch))

invoke:

(def aa "D:\\Users\\input")
(let [ch (read-dir aa)]
  (loop []
    (when-let [line (<!! ch )]
      (println line)
      (recur))))

================

reify the Iterable interace, can be used in java.

MyFiles.clj:
(ns user
  (:gen-class :methods [#^{:static true} [readDir [String] Iterable]])
  (:require [clojure.core.async :as async :refer :all 
:exclude [map into reduce merge partition partition-by take]]))

(defn -readDir [dir]
  (def i nil)
  (let [ch (read-dir dir)
        it (reify java.util.Iterator
             (hasNext [this] (alter-var-root #'i (fn [_] (<!! ch))) (not (nil? i)))
             (next [this] i))
        itab (reify Iterable
               (iterator [this] it))]
    itab))

java code:

for (Object line : MyFiles.readDir("/dir")) {
    println(line)
}

score 1 · Answer 3 · answered Apr 14 '13 at 05:42

1

You can use lazy sequences for this, for example, line-seq.

You must be careful, however, that the sequence returned by line-seq (and other functions which return lazy sequences based on some external resource) never would leak out of e.g. with-open scope because after the source is closed, further reading from lazy sequence will cause exceptions.

answered Apr 14 '13 at 05:42

Vladimir Matveev

120,085
34
287
296

1

But all the computed parts of lazy sequences will stay in memory, won't they? – missingfaktor Apr 14 '13 at 08:49
@missingfaktor: If you don't end up with `holding the head` problem with sequences, you will be safe .. http://stackoverflow.com/questions/2214258/holding-onto-the-head-of-a-sequence – Ankur Apr 14 '13 at 09:25
@missingfaktor, no, of course, they won't. Lazy sequences are sequences, after all, and sequences are singly-linked. So, as Ankur said, if you won't hold the head of the sequence, it will be garbage-collected. – Vladimir Matveev Apr 14 '13 at 10:52

Clojure way of reading large files and transforming data therein

3 Answers3

================

Linked