Encoding in python with lxml - complex solution

Question

I need to download and parse webpage with lxml and build UTF-8 xml output. I think schema in pseudocode is more illustrative:

from lxml import etree

webfile = urllib2.urlopen(url)
root = etree.parse(webfile.read(), parser=etree.HTMLParser(recover=True))

txt = my_process_text(etree.tostring(root.xpath('/html/body'), encoding=utf8))


output = etree.Element("out")
output.text = txt

outputfile.write(etree.tostring(output, encoding=utf8))

So webfile can be in any encoding (lxml should handle this). Outputfile have to be in utf-8. I'm not sure where to use encoding/coding. Is this schema ok? (I cant find good tutorial about lxml and encoding, but I can find many problems with this...) I need robust solution.

Edit:

So for sending utf-8 to lxml I use

        converted = UnicodeDammit(webfile, isHTML=True)
        if not converted.unicode:
            print "ERR. UnicodeDammit failed to detect encoding, tried [%s]", \
                ', '.join(converted.triedEncodings)
            continue
        webfile = converted.unicode.encode('utf-8')

score 19 · Accepted Answer · edited Oct 16 '12 at 16:16

19

lxml can be a little wonky about input encodings. It is best to send UTF8 in and get UTF8 out.

You might want to use the chardet module or UnicodeDammit to decode the actual data.

You'd want to do something vaguely like:

import chardet
from lxml import html
content = urllib2.urlopen(url).read()
encoding = chardet.detect(content)['encoding']
if encoding != 'utf-8':
    content = content.decode(encoding, 'replace').encode('utf-8')
doc = html.fromstring(content, base_url=url)

I'm not sure why you are moving between lxml and etree, unless you are interacting with another library that already uses etree?

edited Oct 16 '12 at 16:16

the Tin Man

158,662
42
215
303

answered Apr 22 '10 at 06:21

Ian Bicking

9,762
6
33
32

Unicode Dammit seems good. And about etree you are right, I've remove it from code. – Vojta Rylko Apr 23 '10 at 14:50
2

Why not pass directly the decoded string (unicode object) to html.fromstring(), instead of re-encoding it to utf-8?? – lajarre Dec 10 '12 at 18:53
1

I can't remember what specifically was the motivation two and a half years ago, but I do vaguely remember that lxml didn't like Unicode input in some cases. There's a very good chance that whatever that problem was it got fixed, so it's probably better to ignore that part now. libxml2 (which powers lxml) does like UTF-8 input though, so if you were very performance sensitive you might want to avoid decoding that encoding in particular. – Ian Bicking Dec 11 '12 at 20:25

score 2 · Answer 2 · edited Dec 30 '15 at 02:49

lxml encoding detection is weak.

However, note that the most common problem with web pages is the lack of (or the existence of incorrect) encoding declarations. It is therefore often sufficient to only use the encoding detection of BeautifulSoup, called UnicodeDammit, and to leave the rest to lxml's own HTML parser, which is several times faster.

I recommend to detect encoding using UnicodeDammit and parse using lxml. Also, you can use http header Content-Type (you need to extract charset=ENCODING_NAME) to detect encoding more precisely.

For this example i'm using BeautifulSoup4 (also you have to install chardet for better autodetection, because UnicodeDammit uses chardet internally):

from bs4 import UnicodeDammit

if http_charset == "":
    ud = UnicodeDammit(content, is_html=True)
else:
    ud = UnicodeDammit(content, override_encodings=[http_charset], is_html=True)
root = lxml.html.fromstring(ud.unicode_markup)

OR, to make previous answer more complete, you can modify it to:

if ud.original_encoding != 'utf-8':
    content = content.decode(ud.original_encoding, 'replace').encode('utf-8')

Why this is better than simple using chardet?

You do not ignore Content-Type HTTP header

Content-Type:text/html; charset=utf-8
You do not ignore http-equiv meta tag. Example:

... http-equiv="Content-Type" content="text/html; charset=UTF-8" ...
On top of this, you are using power of chardet, cjkcodecs and iconvcodec codecs and many more.

Encoding in python with lxml - complex solution

2 Answers2

Linked