6

I need to implement lazy loading in Mathematica. I have a 600 Mb CSV text file which I need to process. This file contains a lot of duplicated records:

1;0;0;13;6
1;0;0;13;6
..........
2;0;0;13;6
2;0;0;13;6
..........
etc.

So instead of loading them all into memory, I'd like to create a list containing records and the number of times this record was encountered in the file:

{{10000,{1,0,0,13,6}}, {20000,{2,0,0,13,6}}, ...}

I couldn't find a way to do it with Import function. I'm looking for something like 

Import["my_file.csv", "CSV", myProcessingFunction]

where myProcessingFunction will take one record at a time and create a dataset. Is it possible to do this with Import or any other Mathematica function?

Max
  • 19,654
  • 13
  • 84
  • 122
  • In case you don't know, this is called Run Length Encoding. There are several neat Mathematica algorithms for implementing it. – Tim Nov 26 '10 at 15:10
  • 2
    @Tim Kemp: before running an algorithm I need to load the data first. The size of the data does not allow it to be loaded at once. My question is about how to load the data line by line, rather than about how to group the records. Also RLE algorithms preserve the sequence of records. I don't need this. I only need the counts. For instance AAABBBBAAAA after RLE becomes 3A,4B,4A. I need 7A, 4B. – Max Nov 26 '10 at 20:51

4 Answers4

2

I think you want the Read[] function.

High Performance Mark
  • 77,191
  • 7
  • 105
  • 161
2

Perhaps there are better alternatives than Mathematica for doing this.

A small awk script:

 {a[$0]++}  
 END { ... print loop ... }

will accumulate the repeated records. Of course you may suffer overflows depending on the number of distinct records.

Or sort the file first and the counting will not overflow. In awk, the non-overflows program may be something like;

 BEGIN{ p =""; i=0}

 {if (($0 != p) &&  (i != 0) ) {print $0,i ; p =$0; i=0; next}}

 {i++; p = $0}

Perhaps Perl is better, but I'm old fashioned.

HTH!

Dr. belisarius
  • 60,527
  • 15
  • 115
  • 190
2

If it were me, I'd probably do this using unix sort and uniq, but since you ask about Mathematica.... I'd use ReadList[] to read blocks of lines, and define downvalues to find the unique strings an keep track of how many we've seen before.

(* Create some test data *)
Export["/tmp/test.txt", Flatten[{Range[1000], Range[1000]}], "Lines"];

countUniqueLines[file_String, blockSize_Integer] := Module[{stream, map, block, keys, out}, 
    map[_]:=0;
    stream = OpenRead[file];
    CheckAbort[While[(block=ReadList[stream, String, blockSize])=!={}, 
        (map[#]=map[#]+1)& /@ block;];, Close[stream];Clear[map]];
    Close[stream];
    keys = Cases[DownValues[map][[All, 1, 1, 1]], _String];
    out = {#, map[#]}& /@ keys;
    Clear[map];
    out
]

countUniqueLines["/tmp/test.txt", 500]


(* Alternative implementation if you have a little more memory *)
Tally[Import["/tmp/test.txt", "Lines"]]
Joshua Martell
  • 7,074
  • 2
  • 30
  • 37
  • +1, definitely the approach I'd take. Three things. The while loop needs to be wrapped in a `CheckAbort`, otherwise the stream will be left open when. This is vital when dealing with a large file. Secondly, the semicolon must be included in `WordSeparators`, or it will generate a lot of errors with a CSV file. Lastly, how well does it behave with a 600 mb file? – rcollyer Nov 27 '10 at 04:42
  • @rcollyer, I added the CheckAbort handling. This function processes lines only; it doesn't separate the fields which saves some memory. It should be as simple as changing ReadList to break on ";", though. – Joshua Martell Nov 27 '10 at 15:42
0

I would recommend you to consider loading it first into a database system like MySQL and then you can access it from Mathematica using the DatabaseLink.

gdelfino
  • 11,053
  • 6
  • 44
  • 48