How to parse broken HTML with LXML

Question

I'm trying to parse a broken HTML with LXML parser on python 2.5 and 2.7

Unlike in LXML documentation (http://lxml.de/parsing.html#parsing-html) parsing a broken HTML does not work:

from lxml import etree
import StringIO
broken_html = "<html><head><title>test<body><h1>page title</h3>"
parser = etree.HTMLParser()
tree   = etree.parse(StringIO.StringIO(broken_html))

Result:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "lxml.etree.pyx", line 2954, in lxml.etree.parse (src/lxml/lxml.etree.c:56220)
  File "parser.pxi", line 1550, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:82482)
  File "parser.pxi", line 1578, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:82764)
  File "parser.pxi", line 1457, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:81562)
  File "parser.pxi", line 965, in lxml.etree._BaseParser._parseDoc (src/lxml/lxml.etree.c:78232)
  File "parser.pxi", line 569, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:74488)
  File "parser.pxi", line 650, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:75379)
  File "parser.pxi", line 590, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:74712)
lxml.etree.XMLSyntaxError: Opening and ending tag mismatch: h1 line 1 and h3, line 1, column 50

score 28 · Accepted Answer · answered Oct 01 '13 at 14:04

Don't just construct that parser, use it (as per the example you link to):

>>> tree = etree.parse(StringIO.StringIO(broken_html), parser=parser)
>>> tree
<lxml.etree._ElementTree object at 0x2fd8e60>

Or use lxml.html as a shortcut:

>>> from lxml import html
>>> broken_html = "<html><head><title>test<body><h1>page title</h3>"
>>> html.fromstring(broken_html)
<Element html at 0x2dde650>

score 12 · Answer 2 · edited Jul 25 '20 at 01:16

12

lxml allows you load a broken xml by creating a parser instance with recover=True

etree.HTMLParser(recover=True)

You could use the same technique when creating the parser.

edited Jul 25 '20 at 01:16

jimh

1,651
2
15
28

answered Nov 07 '14 at 03:18

Jerome Anthony

7,823
2
40
31

2

This answer totally helped me, but note that this question was about `html`, which is usually not XML by design. – mlissner Jun 23 '16 at 00:09

score 9 · Answer 3 · answered Oct 01 '13 at 14:07

You might try to use lxml.html instead

>>> import lxml.html
>>> broken_html = "<html><head><title>test<body><h1>page title</h3>"
>>> root = lxml.html.fromstring(broken_html)
>>> lxml.html.tostring(root)
'<html><head><title>test</title></head><body><h1>page title</h1></body></html>'

How to parse broken HTML with LXML

3 Answers3