perl out of memory message processing just 64 XML file each of 2MB - unix

Question

I tried globalising variables and undef , increasing data segment space in unix , localising variable , but still getting the same error. I need to process around 750 files .Can anyone help? Thanks. I know reading the entire file into string may be a problem. But I am not sure of anyother ways. But still as i declare the string as global and making it ="" . shoulnd tht release memory in next iterations ?

foreach my $file_name (@dir_contents) 
{

if(-f "rawdata/$file_name")
{
$xmlres="";
eval {

while(<FILE>)
{
    $xmlres.=$_;
}
close FILE;


 ***$doc=$parser->parsestring($xmlres);***  
foreach my $node($doc->getElementsByTagName("nam1"))
{
    foreach my $tnode($node->getElementsByTagName(("name2")))
    {
        //processing
    }
}
}

} }

This can't be complete code, it doesn't compile for me, too many errors. `switch/case`, really? Does that even work? I thought the keywords were `given` and `when`. — TLP, Jul 12 '11 at 12:06
Another "really?" is the use of `eval` here on a big block of code. Use a subroutine. — David Hammen, Jul 12 '11 at 12:32
Agree with David. This code is a mess. Too many variables, too many globals, too many weird things. You are setting global variables inside the sub, using `eval` for some reason, using undef on global variables instead of restricting their scope. And you do not seem to be using strict, or else `$src` is defined elsewhere. My advise is: Toss this code out and write new one, using proper programming techniques. — TLP, Jul 12 '11 at 12:40

score 3 · Accepted Answer · answered Jul 12 '11 at 13:45

First of all, the style comments are useful and correct, and would help. However, if you need to process 1.5Gb of XML, you're going to need to manage memory a little bit better.

XML::DOM doesn't automatically free space it used. This is a sign of its age, and newer modules manage memory much better, and tend to do this automatically (I also use XML::LibXML, which does this, and I'd also recommend it highly).

Mainly, you need to call the dispose method to clean out a DOM tree when you have finished with it. This is fairly clear in the pod synopsis for XML::DOM. Just calling it may be enough to get your memory issues resolved. (Technically, DOM trees tend to contain cyclical references, and these are not automatically managed in simple referencing counting garbage collection. Perl has used weak references to assist, but it looks this hasn't been integrated in XML::DOM fully. Simply unreferencing the tree is not enough.)

I'd certainly look to improve style elsewhere. Some other style issues; I'd try Try::Tiny to handle the eval {}, as you seem to be using it mainly for exception handling. Also, several bad experiences have taught me that using a solid date/time parser is always a good idea. I use the ones in DateTime::Format::*. There are many odd cases in date and time parsing, and this will save you lines of code and make the handling more reliable.

score 2 · Answer 2 · answered Jul 12 '11 at 11:12

XML::DOM is old and limited (not to mention that I don't think it's maintained any more). Try XML::LibXML, which is very similar (it also implements a DOM), except faster, more memory-frugal, more powerful (full XPath implementation...), maintained...

perl out of memory message processing just 64 XML file each of 2MB - unix

2 Answers2