6

I create a XML String on the fly (NOT reading from a file). Then I use Cocoon 3 to transform it via FOP to a PDF. Somewhere in the middle Xerces runs. When I use the hardcoded stuff everything works. As soon as I put a german Umlaut into the database and enrich my xml with that data I get:

Caused by: org.apache.cocoon.pipeline.ProcessingException: Can't parse the XML string.
at org.apache.cocoon.sax.component.XMLGenerator$StringGenerator.execute(XMLGenerator.java:326)
at org.apache.cocoon.sax.component.XMLGenerator.execute(XMLGenerator.java:104)
at org.apache.cocoon.pipeline.AbstractPipeline.invokeStarter(AbstractPipeline.java:146)
at org.apache.cocoon.pipeline.AbstractPipeline.execute(AbstractPipeline.java:76)
at de.grobmeier.tab.webapp.modules.documents.InvoicePipeline.generateInvoice(InvoicePipeline.java:74)
... 87 more

Caused by: com.sun.org.apache.xerces.internal.impl.io.MalformedByteSequenceException: Invalid byte 1 of 1-byte UTF-8 sequence.
    at com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.invalidByte(UTF8Reader.java:684)
    at com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.read(UTF8Reader.java:554)

I have then debugged my app and found out, my "Ä" (which comes frome the database) has the byte value of 196, which is C4 in hex. This is what I have expected according to this: http://www.utf8-zeichentabelle.de/

I do not know why my code fails.

I have then tried to add a BOM manually, like that:

byte[] bom = new byte[3];
bom[0] = (byte) 0xEF;
bom[1] = (byte) 0xBB;
bom[2] = (byte) 0xBF;
String myString = new String(bom) + inputString;

I know this is not exactly good, but I tried it - of course it failed. I have tried to add a xml header in front:

<?xml version="1.0" encoding="UTF-8"?>

Which failed too. Then I combined it. Failed.

After all I tried something like that:

xmlInput = new String(xmlInput.getBytes("UTF8"), "UTF8");

Which is doing nothing in fact, because it is already UTF-8. Still it fails.

So... any ideas what I am doing wrong and what Xerces is expecting from me?

Thanks Christian

Christian
  • 7,062
  • 5
  • 36
  • 39
  • Agreed, but it does not help me. Because the problematic String which comes from the database is created from my ORM layer. In addition, it has 0xC4 which should do fine, right? – Christian Dec 12 '11 at 08:58
  • I use MySQL, table and columns are encoded with utf8_general_ci. I have added useUnicode=true&characterEncoding=utf8 to my jdbc connection. – Christian Dec 12 '11 at 09:12
  • It might not be a good idea to specify those parameters on the JDBC connection when you do happen to connect to a DB with a different encoding - only use it when autodetection goes wrong. What do you use to write the data, and is this a BLOB or a VARCHAR column? – JBert Dec 12 '11 at 09:34
  • My tool is not so generic, it will only connect to the db with a encoding i put. In addition I actually had problems with not specifying it. The data I write comes from a webpage, which has itself UTF-8 encoded: <%@ page contentType="text/html; charset=UTF-8" pageEncoding="UTF-8" %> . Then it goes directly into Struts. – Christian Dec 12 '11 at 09:40
  • I meanwhile found out, that I can prove I use 100% utf-8. While the whole trial/error phase I looked at the string char in the debugger, which are somehow using 196, instead of returning the actual bytes. Now I have identified the correct sequence of bytes. Thanks for your comments, I have learned something about unicode/utf-8 now. Anyway, my problem was actually in Cocoon 3, where I need to do new XMLGenerator(xmlInput.getBytes("UTF-8"), "UTF-8") instead of just giving the String. – Christian Dec 12 '11 at 10:35
  • Great! I've added your findings to my answer and did some more detective work so that you can close this question. – JBert Dec 12 '11 at 11:35
  • Great you have found out the Cocoon problem. Therefore you will get the "thanks" flag. I will forward your post to my Cocoon-friends. Cheers – Christian Dec 12 '11 at 11:44

3 Answers3

13

If your database contains only a single byte (with value 0xC4) then you aren't using UTF-8 encoding.

The character "LATIN CAPITAL LETTER A WITH DIAERESIS" has a code-point value U+00C4, but UTF-8 can't encode that in a single byte. If you check the third column "UTF-8 (hex.)" on UTF8-zeichentabelle.de you'll see that UTF-8 encodes that as 0xC3 84 (two bytes).

Please read Joel's article "The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)" for more info.

EDIT: Christian found the answer himself; turned out it was a problem in the Cocoon 3 SAX component (I guess it's the alpha 3 version). It turns out that if you pass an XML as a String into the XMLGenerator class, something will go wrong during SAX parsing causing this mess.

I looked up the code to find the actual problem in Cocoon-stax:

if (XMLGenerator.this.logger.isDebugEnabled()) {
    XMLGenerator.this.logger.debug("Using a string to produce SAX events.");
}
XMLUtils.toSax(new ByteArrayInputStream(this.xmlString.getBytes()), XMLGenerator.this.getSAXConsumer();

As you can see, the call getBytes() will create a Byte array with the JRE's default encoding which will then fail to parse. This is because the XML declares itself to be UTF-8 whereas the data is now in bytes again, and likely using your Windows codepage.

As a workaround, one can use the following:

new org.apache.cocoon.sax.component.XMLGenerator(xmlInput.getBytes("UTF-8"),
       "UTF-8");

This will trigger the right internal actions (as Christian found out by experimenting with the API).

I've opened an issue in Apache's bug tracker.

EDIT 2: The issue is fixed and will be included in an upcoming release.

JBert
  • 3,311
  • 24
  • 37
2

The C4 you see on that page refers to the unicode code point, U+00C4. The byte sequence used to represent such a code point in UTF-8 is NOT "\xC4". What you want is what's in the UTF-8 (hex.) column, namely "\xC3\x84".

Therefore, your data is not in UTF-8.

You can read about how data is encoded in UTF-8 here.

Artefacto
  • 96,375
  • 17
  • 202
  • 225
0

I'm running Windows 7 with TextPad as a text editor for manually building the xml data file. I was getting the MalformedByteSequenceException. My spec in the xml file was UTF-8. After poking around, I found that my editor had a tool "Tools ... Convert to DOS". I did that, re-saved the file, and the exception went away and my code ran fine.

I then looked at the default encoding for that file type in my editor. It was ASCII, though when I changed the xml encoding parameter to ASCII, I got another different MalformedByteSequenceException.

So on Windows systems, you might try keeping the xml encoding to UTF-8, but save the file encoded DOS. I did not dig any further as to why this works.

Robert
  • 5,278
  • 43
  • 65
  • 115
NeilW
  • 1
  • 1