1

I am using the roo gem to parse data in a CSV. The CSV is around 400MB in size however, and when I attempt to parse it with this loop...

(2..sheet.last_row).collect do |i|
   # parsing code...
end

It will error out with [FATAL] failed to allocate memory. I was going to break it up with smaller loops until it reaches the last_row, but just asking roo for the last_row will cause the error -- so I can't really tackle it that way.

daveomcd
  • 6,367
  • 14
  • 83
  • 137
  • 1
    You need to process it one line at a time, rather than loading the whole lot into a giant array and then iterating over the array. I use FasterCSV which has a "process a file one line at a time" method. Looking at roo on https://github.com/roo-rb/roo i can't see this functionality. – Max Williams Jun 15 '15 at 12:22
  • @MaxWilliams, thanks max. I'm also seeing that the Roo gem for this is limiting. It appears it uses the ruby CSV library which from what I read is pretty outdated. I'm in the middle of exploring the `smarter-csv` gem and it appears to be handling the issue. Just a matter of redoing my work. – daveomcd Jun 15 '15 at 12:24

0 Answers0