Having trouble locating memory leak in Perl

Question

I'm experiencing a memory leak (I think) in this snippet of code:

if (ref($init{'time-layout'}) eq 'HASH') {
    my %time= % {$init{'time-layout'}};
    my @times= @ {$time{'start-valid-time'}};
    if (ref($init{parameters}) eq 'HASH') {
        %param= % {$init{parameters}};
        if (ref($param{'wind-speed'}) eq 'HASH') {
            %wind= % {$param{'wind-speed'}};
            @windvalue= @ {$wind{value}};
            %temp= % {$param{'temperature'}};
            @tempvalue = @ {$temp{value}};
            %wdir= % {$param{'direction'}};
            @wdirvalue = @ {$wdir{value}};
            %hum= % {$param{'humidity'}};
            @humvalue = @ {$hum{value}};
        undef %wind;
        undef %temp;
        undef %wdir;
        undef %hum;}
    undef %param;}

This is from a sub, where the input comes from XMLin() (from XML::Simple). Specifically the input is (if it matters)

http://graphical.weather.gov/xml/sample_products/browser_interface/ndfdXMLclient.php?    lat=$lat&lon=$lon&product=time-series&begin=2004-01-01T00:00:00&end=2013-04-20T00:00:00&temp=temp&wspd=wspd&rh=rh&wdir=wdir";

where $lat and $lon are latitude and longitude.

The code takes a couple hours, to run plugging data into a SQL server, and well before it's done the instance of Perl is taking several GBs of RAM or more. I'm sure this is simple for someone but I'm, at best, a novice in Perl.

If you are not making changes to data that you want to keep read-only, don't make copies of it, e.g. just say `@windvalue = @{$init{parameters}->{'wind-speed'}->{value}}`. — imran, Apr 03 '13 at 21:09
@imran I made those changes and it may have appeared to help a bit but the other answers about XML::Simple being the problem appear to be correct. — Dean MacGregor, Apr 04 '13 at 00:12
@DeanMacGregor I would recommend using XML::LibXML::Reader module. — , Apr 05 '13 at 08:27

score 3 · Accepted Answer · answered Apr 03 '13 at 22:00

3

A quick duckduckgo search for "xml::simple memory leak" turned up the documentation for Yahoo::Search, which claims:

XML::Simple uses XML::Parser under the hood, and at least on the systems I've tested it, XML::Parser suffers from a crippling memory leak that makes it very undesirable.

It appears possible (perhaps even likely) that the memory leak may not be in your code at all.

answered Apr 03 '13 at 22:00

Dave Sherohman

45,363
14
64
102

I believe it was both because I carried out imran's suggestion which helped a little and I was just finding that my RAM usage was going up each time I pass over my XMLin command. Is this the problem to which you were refering? http://www.perlmonks.org/index.pl?node_id=251630 I hope not its 10 years old. – Dean MacGregor Apr 04 '13 at 00:11

score 3 · Answer 2 · answered Apr 04 '13 at 00:04

3

XML::Simple is very ineffective on parsing large files (for me it eats 3Gb of memory for the 100Mb file). Try to use XML::Twig for your data.

answered Apr 04 '13 at 00:04

gangabass

10,607
2
23
35

score 1 · Answer 3 · edited May 23 '17 at 12:14

In addition to imran's comment,

With so many XML cpan modules, it is very hard to decide which one should be used.

Looking at answers from many and reading from web, XML::LibXML looks a preferred module, rather more specific XML::LibXML::Reader for large files. Many of them haven't recommended using XML::Simple and says XML::Simple is good for small files and have issues when it comes to large xml files. XML::Twig is better(from it's documentation), but even that is having memory leak problems also. (refer XML::Twig faq)

You wanna also refer perl-xml-faq

Having trouble locating memory leak in Perl

3 Answers3