1

I have a task that requires me to load log files that is too big for my memory to let it load at once. Therefore I need way to load the file by different chunks..

I know that:

-11!(n; filename)

loads first n chunk from a file, but how do I load the rest of the chunks??

Thanks!

chrisaycock
  • 36,470
  • 14
  • 88
  • 125
user1948847
  • 955
  • 1
  • 12
  • 27

1 Answers1

3

Your log file probably calls the function upd if it is a typical tickerplant setup. Override that function to ignore the initial values on the second run, without inserting them into the table allowing you to split the loading into parts.

upd:{i+:1; if[i within 100000 200000; ignore x; storeSomewhere x]};

Then play the whole file -11!. Control what portions are actually stored by changing the condional in the if clause.

Thomas Smyth - Treliant
  • 4,993
  • 6
  • 25
  • 36
John at TimeStored
  • 1,305
  • 7
  • 9
  • Thanks for the answer! Just a couple more thing, for overriding the upd, which function should I use? And after I override the upd, should i just use -11!(n; filename) to get the next n lines? – user1948847 Mar 15 '13 at 12:56