-1

I have a parser class that is written for parsing xml files and import data to DB. During parsing I collect dozen of objects for import(avg. 60_000) into one array and insert all of them at one time. After operation finished RAM doesn't rollback to previous state. So, after few parsing operations my RAM totally runs out. I try to figure out why memory doesn't clean up from finished process.

Stack:

  • ruby 2.2.1
  • rails 4.2.4
  • nokogiri 1.6.6.2
  • activerecord 4.2.4
  • activerecord-import 0.10.0

I'm inexperienced in this field and don't know where to start. So, I appreciate any help.

Igor
  • 13
  • 6
  • Please read "[mcve]" and http://tinyurl.com/stack-hints. We need to be able to see the minimal code you're using that duplicates the problem. As is you're asking us to guess what you wrote, which will do no good as we can guess some pretty wild things. Without your help we'll waste our time, taking away from helping others, so you need to help us help you. – the Tin Man Jan 25 '16 at 20:51
  • Also, we need the *minimal* example of the code necessary to demonstrate the problem *in the question itself*. Links to your full code do us no good because we have to wade through everything that isn't related; The idea is you help us help you by testing, reducing, and then supplying the bare code that shows the problem. In that process you'll often discover the problem and end up fixing it yourself. – the Tin Man Jan 26 '16 at 17:37

1 Answers1

0

I hope you are clearing your array after each and every operation.

Moreover, why are you not parsing and storing data in smaller batches?

This scenario has been logged as a bug in "Objects in large arrays are memory leaked in Ruby".

You can find a similar issue in "Big array and memory".

You can try to force garbage collection after the operation. See "leaks in Ruby".

Community
  • 1
  • 1
Ankit Tripathi
  • 405
  • 4
  • 13
  • You should finish the startup tutorials for Stack Overflow, and then you'll have plenty of points allowing you to do normal things. Please work on your formatting also. Readability is important. – the Tin Man Jan 25 '16 at 20:18