1

I have a very large file and each line can be parsed into a view object. However, I want to return a iterator[A] instead of collection, so it can have better memory characteristics for the large file parsing.

factory.createContainer(line: String): Foo = .......

def parse: Iterator[Foo] = {
io.linesR("src/test/resources/largedummy.txt")
    .map(line => factory.createContainer(line))
    .to(.......)

    // I am not sure how can write it in here to return a Iterator[Foo]
}

Many thanks in advance

Xiaohe Dong
  • 4,953
  • 6
  • 24
  • 53
  • 4
    It'd be better to accomplish what you're trying to do in the context of Scalaz streams—iterators are fundamentally not functional, and mixing up the two concepts here is likely to be messy or confusing. – Travis Brown Sep 16 '14 at 12:43
  • Your point makes sense, I hesitated before that I should ask this question or not, runlog.run would map to Seq[A], I thought it can return a iterator, but maybe it is not a good idea. – Xiaohe Dong Sep 16 '14 at 13:12
  • Instead of using `runLog`, can you move whatever operation you were going to do with the resulting iterator or collection into the context of Scalaz stream? – Travis Brown Sep 16 '14 at 13:42
  • Yes, I can, e.g. The method would need to accept a trait like service: Service[T] {def perform: T } to let users to implement it, and this service would be used in the context. – Xiaohe Dong Sep 16 '14 at 14:55
  • However, I am not convinced that, if Stream is designed that we cannot step twice into the same stream, why it can support to transform Seq very well but not Iterator. I am sorry this may be beyond this question, but this is still my confusion. – Xiaohe Dong Sep 16 '14 at 15:25
  • @Cloudtech It is not necessary to transform to an `iterator`. because a `Process` itself is lazy, can do `map`/`scan` and much more . – jilen Sep 16 '14 at 23:49
  • I think I shouldnt use Stream to return an iterator, as you guys mentioned, if there is no reply on this, I will close this at the end of this day. – Xiaohe Dong Sep 17 '14 at 00:42
  • 1
    can you please add the code you want to do assuming the run.runLog WILL return Iterator? I can then give you ideas how to achieve that in streams directly. – Pavel Chlupacek Sep 17 '14 at 04:53

0 Answers0