Cannot seem to render XML or clean characters

Question

I'm trying to render my sitemap.xml using flask, however I cannot seem to clean my input such that the rendering can work. The error is the following:

error on line 23325 at column 83: PCDATA invalid Char value 11

The code I'm using is:

url = prepend + "/explore/"+str(result['id'])+"/"+result['title'].encode('utf-8', 'xmlcharrefreplace').decode('utf-8')

I've also tried:

url = prepend + "/explore/"+str(result['id'])+"/"+result['title'].encode('ascii', 'xmlcharrefreplace').decode('ascii')

Also removing what is Char 11, doesn't seem to work either.

result["title"] = result["title"].replace('&#011;', '')

Is there a smarter way that I can clean any spurious characters or problem solve more accurately?

I've tried to flag the problem. It looks like a linefeed or something along those lines. I've tried before and after the URL is built.

XX XXX XXX XXX  
                Step by step guide

but, I cannot find anything in the python arsenal to remove the linefeed. I'm trying, replace('\r', ''), strip() etc etc, but still this persists.

Which XML parser do you use? Try this to remove whitespaces between elements from XML document: http://stackoverflow.com/questions/3310614/remove-whitespaces-in-xml-string — Viach Kakovskyi, May 09 '16 at 14:18
FWIW, char 11 = \x0b is a vertical tab, not a linefeed which is char 10 = \x0a. — PM 2Ring, May 09 '16 at 15:17

score 4 · Accepted Answer · answered May 09 '16 at 15:17

“Char value 11” (0xB in hex) refers to the vertical tab \v, not the carriage return \r, nor the line feed \n.

The vertical tab is not a valid character in an XML document. It’s so invalid that you can’t even represent it as a character reference like . You have to remove it from the string yourself:

result["title"] = result["title"].replace('\x0b', '')

While you’re at it, you may want to remove other invalid characters as well, as they have a tendency to pop up inscrutably in user-supplied data and bite you in production. I’m not aware of common library functions to do this, so I came up with my own function that makes both XML and HTML5 more or less happy:

import re

def printable(s):
    # Based on `XML 1.0 section 2.2 <https://www.w3.org/TR/xml/#charsets>`_,
    # with the addition of U+0085,
    # which the W3C (Nu) validator also marked as a "forbidden code point".
    # Even with this code, the validator still complains about
    # "Text run is not in Unicode Normalization Form C"
    # and "Document uses the Unicode Private Use Area(s)".
    return re.sub(
        pattern=(u'[\u0000-\u0008\u000B\u000C\u000E-\u001F'
                 u'\u007F-\u009F\uD800-\uDFFF\uFDD0-\uFDEF\uFFFE\uFFFF]'),
        repl=u'\N{REPLACEMENT CHARACTER}',
        string=s
    )

As for xmlcharrefreplace, that’s not what you’re looking for at all. It’s not an escaping mechanism for XML-invalid characters. It’s a way to preserve characters that are outside of the chosen encoding. It would make sense in a situation like this:

>>> print(u'Liberté, égalité, fraternité!'.encode('ascii', 'xmlcharrefreplace'))
Libert&#233;, &#233;galit&#233;, fraternit&#233;!

Cannot seem to render XML or clean characters

1 Answers1