1

I've written an application that parses large XML files in PHP using XMLReader.

Generally, the application works great, EXCEPT when I attempt to read a file that's larger than 2gb.

(I haven't figured out exactly where the cut-off is; it works flawlessly on a 500mb file, but fails on the next largest file I have - 2.5gb).

Specifically, if my code looks like this:

$reader = new XMLReader();
if ($reader->open("big.xml")) {
    echo "Success!";
    $reader->close();
} else {
    echo "Failed!";
}

If I test the large (>2gb) file - I get this:

Warning: XMLReader::open() [xmlreader.open]: Unable to open source data in [php script]

And of course, Failed! is output.

If I try with a smaller (500mb) file - I get only the Succcess! output.

As far as I can tell - there's no difference between the large files that can't be opened and the medium-size files that can be opened (e.g. permissions, valid XML, encoding) EXCEPT the size of the file.

While the size of the file is large - the nodes are all tiny, so I don't think any single node would cause a memory issue.

mattstuehler
  • 9,040
  • 18
  • 78
  • 108
  • 2
    You're running 32-bit PHP, which means the filepointer is a 32-bit signed integer, which gives a limit of 2GB for a filesize. The latest PHP 5.6.0 (64-bit) has fixed this issue – Mark Baker Sep 18 '14 at 15:30

1 Answers1

1

PHP may be running out of memory. Try:

ini_set('memory_limit','256M');
Jake Lee
  • 7,549
  • 8
  • 45
  • 86
  • 1
    This shouldn't be a memory issue as XMLReader is a pull parser – Mark Baker Sep 18 '14 at 15:29
  • Purely anecdotal fix. I had this issue a few years back, this solved it. Seems like someone else had the same problem: http://stackoverflow.com/questions/5128811/how-to-use-xmlreader-domdocument-with-large-xml-file-and-prevent-500-error Agreed though, there *shouldn't* be a memory issue. – Jake Lee Sep 18 '14 at 15:31