python lxml iterparse fails on large files containing namespaces

Question

I'm tryint to parse large file (>100mb) as described at http://effbot.org/zone/element-iterparse.htm#incremental-parsing

But if file contains namespaces, lxml fails with error

lxml.etree.XMLSyntaxError: Namespace default prefix was not found

It works fine if I remove elem.clear(), but uses a lot of memory. Example of xml file

<?xml version="1.0" encoding="utf-8" ?>
<feed xmlns="NS">
  <offer>
    <type>type1</type>
    <name>name1</name>
  </offer>
</feed>

lxml version is 3.2.0, because new versions segfaults after end of parsing

Could you provide sample code? – maxbublis Mar 25 '14 at 16:19 — maxbublis, Mar 25 '14 at 16:19
I've tried lxml>3.3 and all is OK now – vitalii Jan 29 '15 at 09:14 — vitalii, Jan 29 '15 at 09:14

score 0 · Answer 1 · edited May 23 '17 at 12:28

0

Did you read this? In my experience with 100MB+ files you are in over 2GB ram usage memory (eg with my 160MB ones I'm up to 4.5GB) Are you using 64 bit python?

edited May 23 '17 at 12:28

Community

1
1

answered Mar 25 '14 at 15:25

Stabby

109
7

python lxml iterparse fails on large files containing namespaces

1 Answers1