1

I'm having trouble dealing with Charsets while parsing and rendering a page using the JSoup library. here is an example of the page it renders:

http://dl.dropbox.com/u/13093/charset-problem.html

As you can see, where there should be ' characters, ? is being rendered instead (even when you view the source).

This page is being generated by downloading a web page, parsing with JSoup, and then re-rendering it again having made some structural changes.

I'm downloading the page as follows:

final Document inputDoc = Jsoup.connect(sourceURL.toString()).get();

When I create the output document I do so as follows:

outputDoc.outputSettings().charset(Charset.forName("UTF-8"));
outputDoc.head().appendElement("meta").attr("charset", "UTF-8");
outputDoc.head().appendElement("meta").attr("http-equiv", "Content-Type")
            .attr("content", "text/html; charset=UTF-8");

Can anyone offer suggestions as to what I'm doing wrong?

edit: Note that the source page is http://blog.locut.us/ and as you'll see, it appears to render correctly

sanity
  • 35,347
  • 40
  • 135
  • 226
  • It sure looks like an encoding issue. Perhaps at the server? If you connect directly and look at the returned stream, does the text still have the "?" problem? – Ted Hopp Aug 01 '11 at 16:38
  • The source page is http://blog.locut.us/ and it appears to render correctly :-/ – sanity Aug 01 '11 at 16:50

2 Answers2

4

The question marks are typical whenever you write characters to the outputstream of the response which are not covered by the response's character encoding. You seem to be relying on the platform default character encoding when serving the response. The response Content-Type header of your site also confirms this by a missing charset attribute.

Assuming that you're using a servlet to serve the modified HTML, then you should be using HttpServletResponse#setCharacterEncoding() to set the character encoding before writing the modified HTML out.

response.setCharacterEncoding("UTF-8");
response.getWriter().write(html);
BalusC
  • 1,082,665
  • 372
  • 3,610
  • 3,555
0

The problem is most likely in reading the input page, you need to have the correct encoding for the source too.

Leonard Brünings
  • 12,408
  • 1
  • 46
  • 66