5

I have a text file of size 100-200 GB. So I wish to store in a compressed format (such as zip). However, I need to process it one line at a time due to its size. Though it is straightforward to read a text file one line at a time with io.Source.fromFile(fileName).getLines, but that is only for unzipped files.

Is there some efficient way to read a compressed file in scala line-by-line? I couldn't find any examples but a closer implementation I saw was here but it loads file into memory. Unlike examples that are generally given that work with zip archive, I need to process only one text file that is compressed. I would be grateful for any pointers or leads.

xinit
  • 147
  • 9
Quiescent
  • 1,088
  • 7
  • 18

2 Answers2

5

If the file is Gzipped, java's GzipInputStream gives you streaming access:

   val lines: Iterator[String] = Source
     .fromInputStream(new GzipInputStream(new FileInputStream("foo.gz")))
     .getLines

If it is a zip archive as your question suggests, that's more complicated. Zip archives are more like folders than individual files. You'd have to read the table of content first, and then scan through the entries to find one you want to read (or to read all of them). Something like this

Dima
  • 39,570
  • 6
  • 44
  • 70
  • 1
    Thank you very much, it was very helpful. I had to just change the case to `GZIPInputStream` to make it work. – Quiescent May 25 '21 at 07:19
3

Consider better-files which gives an Iterator into the compressed file

import better.files._
File("foo.gz").newInputStream.asGzipInputStream().lines
// : Iterator[String]

so you could process the file line by line without loading it whole into memory.

Mario Galic
  • 47,285
  • 6
  • 56
  • 98
  • Thank you for the reply. Dima's suggestion was with in-built library so it was a little easier to use, but I was not aware of better-files which is very versatile and powerful. Learnt something very useful. – Quiescent May 25 '21 at 07:22