Is ColdFusion not capable of returning a valid UTF-8 JSON object?

Question

I have a CFC script, which returns a struct, converted into JSON format. This is the method declaration:

<CFFUNCTION name="getJSON" returntype="Struct" returnformat="JSON" output="no" access="remote">

The output looks just fine, but it's not in UTF-8.

I have tried this:

<CFCONTENT type="application/json; charset=utf-8">
<CFPROCESSINGDIRECTIVE pageEncoding="utf-8">
<CFSCRIPT>
SetEncoding("url", "utf-8");
SetEncoding("form", "utf-8");
</CFSCRIPT>

And I saved the .cfc file as UTF-8 with BOM.

I want to use this as the datasource for a Jasper report, but I always get this error message:

Invalid UTF-8 middle byte 0x74
at [Source: java.io.ByteArrayInputStream@6c2071a7; line: 1, column: 745]

The JSON data contains german umlauts. First one of them is at this position: "line: 1, column: 745".

The error occurs when creating the datasource. InputStream is not in UTF-8.

jsonUrl = getParam('JSON_URL', ReportParamArray);
inputStream = CreateObject("java", "java.net.URL").init(jsonUrl).openConnection().getInputStream();
jasperPrint = jasFillManager.fillReport(jasReport, parameters,
                CreateObject("java", "net.sf.jasperreports.engine.data.JsonDataSource").init(inputStream));

This is the full Stacktrace:

    Error evaluating expression :
Source text : ((net.sf.jasperreports.engine.data.JsonDataSource) $P{REPORT_DATA_SOURCE}).subDataSource("ARBEITSRAPPORT")
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1284):1284
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:588):588
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidOther(UTF8StreamJsonParser.java:2832):2832
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidOther(UTF8StreamJsonParser.java:2839):2839
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._decodeUtf8_3fast(UTF8StreamJsonParser.java:2661):2661
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:1962):1962
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString(UTF8StreamJsonParser.java:1911):1911
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:276):276
at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:203):203
at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:58):58
at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15):15
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:2580):2580
at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:1500):1500
at net.sf.jasperreports.engine.data.JsonDataSource.(JsonDataSource.java:101):101
at net.sf.jasperreports.engine.data.JsonDataSource.subDataSource(JsonDataSource.java:448):448
at rechnungen_1441802385613_104250.evaluate(rechnungen_1441802385613_104250:415):415
at net.sf.jasperreports.engine.fill.JREvaluator.evaluate(JREvaluator.java:231):231
at net.sf.jasperreports.engine.fill.JRCalculator.evaluate(JRCalculator.java:591):591
at net.sf.jasperreports.engine.fill.JRCalculator.evaluate(JRCalculator.java:559):559
at net.sf.jasperreports.engine.fill.JRFillElement.evaluateExpression(JRFillElement.java:1001):1001
at net.sf.jasperreports.engine.fill.JRFillSubreport.evaluateSubreport(JRFillSubreport.java:392):392
at net.sf.jasperreports.components.table.fill.FillTableSubreport.evaluateSubreport(FillTableSubreport.java:101):101
at net.sf.jasperreports.components.table.fill.FillTable.evaluate(FillTable.java:117):117
at net.sf.jasperreports.engine.fill.JRFillComponentElement.evaluate(JRFillComponentElement.java:108):108
at net.sf.jasperreports.engine.fill.JRFillElementContainer.evaluate(JRFillElementContainer.java:259):259
at net.sf.jasperreports.engine.fill.JRFillBand.evaluate(JRFillBand.java:456):456
at net.sf.jasperreports.engine.fill.JRVerticalFiller.fillColumnBand(JRVerticalFiller.java:2044):2044
at net.sf.jasperreports.engine.fill.JRVerticalFiller.fillDetail(JRVerticalFiller.java:778):778
at net.sf.jasperreports.engine.fill.JRVerticalFiller.fillReportStart(JRVerticalFiller.java:288):288
at net.sf.jasperreports.engine.fill.JRVerticalFiller.fillReport(JRVerticalFiller.java:151):151
at net.sf.jasperreports.engine.fill.JRBaseFiller.fill(JRBaseFiller.java:932):932
at net.sf.jasperreports.engine.fill.JRBaseFiller.fill(JRBaseFiller.java:864):864
at net.sf.jasperreports.engine.fill.JRFiller.fill(JRFiller.java:88):88
at net.sf.jasperreports.engine.JasperFillManager.fill(JasperFillManager.java:653):653
at net.sf.jasperreports.engine.JasperFillManager.fillReport(JasperFillManager.java:969):969
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method):-2
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source):-1
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source):-1
at java.lang.reflect.Method.invoke(Unknown Source):-1
at lucee.runtime.reflection.pairs.MethodInstance.invoke(MethodInstance.java:55):55
at lucee.runtime.java.JavaObject.call(JavaObject.java:234):234
at lucee.runtime.java.JavaObject.call(JavaObject.java:256):256
at lucee.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:742):742
at lucee.runtime.PageContextImpl.getFunction(PageContextImpl.java:1589):1589
at cfdocs.gebman.reports.reportsystemjasper_cfm$cf.call(D:\ICS\Internet\CFDOCS\gebman\reports\reportSystemJasper.cfm:158):158
at lucee.runtime.PageContextImpl.doInclude(PageContextImpl.java:950):950
at lucee.runtime.tag.CFTag.doInclude(CFTag.java:323):323
at lucee.runtime.tag.CFTag.cfmlStartTag(CFTag.java:245):245
at lucee.runtime.tag.CFTag.doStartTag(CFTag.java:177):177
at cfdocs.gebman.reports.rechnungindex_cfm$cf.call(D:\ICS\Internet\CFDOCS\gebman\reports\rechnungIndex.cfm:151):151
at lucee.runtime.PageContextImpl.doInclude(PageContextImpl.java:950):950
at lucee.runtime.listener.ClassicAppListener._onRequest(ClassicAppListener.java:56):56
at lucee.runtime.listener.MixedAppListener.onRequest(MixedAppListener.java:36):36
at lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2257):2257
at lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2224):2224
at lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:456):456
at lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:47):47
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728):728
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305):305
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210):210
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222):222
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123):123
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472):472
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171):171
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99):99
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118):118
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407):407
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004):1004
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589):589
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310):310
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source):-1
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source):-1
at java.lang.Thread.run(Unknown Source):-1

How can I ensure, that the CFC returns valid UTF-8 content?

Where does your content data come from? It's likely that the data is not properly encoded to begin with. — Alex, Sep 04 '15 at 15:59
How many bytes does the character at line 1, column 745 have in your table? — Alex, Sep 04 '15 at 16:09
The column belongs to the returned JSON file. It's a file with one line. It's quite difficult to find out how big this character is in the database.. — Christoph Bühler, Sep 04 '15 at 16:26
JSON file? Do you store the JSON in your database or do you build the struct using data from the query? I'm confused. — Alex, Sep 04 '15 at 16:33
I build the struct from a query. I thought, that you've confused "column: 745" with a database column. — Christoph Bühler, Sep 04 '15 at 21:39
Silly question... but not being very familiar with JasperReports it sounds like you are supplying the cfc url to some sort of datasource class, ie http://server/yourcomponent.cfc?method=getJSON? If so, my question would be how does that class determine whether the content is UTF8? — Leigh, Sep 05 '15 at 17:15
In other words, exactly what operation reports the response is not UTF8? The error snippet shows a ByteArrayInputStream error, but not how it is used in the broader code. — Leigh, Sep 05 '15 at 23:42
@Leigh Thank you for your answer. I have edited the question. — Christoph Bühler, Sep 07 '15 at 06:47
Can you post the full stack trace? I tried a simple test with jasperreports 6.1.0. I created a datasource from an call to a cfc (like above). The test data returned was the characters in your name. Surprisingly I did not get an error on this line: `ds = createObject("java", "net.sf.jasperreports.engine.data.JsonDataSource").init(inputStream);` Perhaps the error is occurring when the datasource is actually used by the jasFillManager, not just when it is created. So I would be interested in seeing the complete stack trace. — Leigh, Sep 08 '15 at 00:48
BTW, here is the test code I used with jasper 6.1.0. https://gist.github.com/anonymous/7cbacb9856e9de8a2b1a — Leigh, Sep 08 '15 at 20:13
@Leigh Thank you for your great effort. I postet the full Stacktrace. — Christoph Bühler, Sep 09 '15 at 06:43
Thanks, but that looks like the error message. IF you have full debugging enabled, the stack trace is a small link at the bottom of the error message. Click the link to expand the trace. The text of the will look [more like this](http://stackoverflow.com/questions/937075/coldfusion-call-to-webservice-gives-org-xml-sax-saxexception). You can also wrap the problem code in a cftry/cfcatch. Then dump the exception details inside the catch clause, ie `` — Leigh, Sep 09 '15 at 12:39
From what I can see, the error is thrown by the jackson parser ie `com.fasterxml.jackson.core.json.UTF8StreamJsonParser`. A quick search turned up [this thread](http://stackoverflow.com/questions/6352861/jackson-jsonparseexception-invalid-utf-8-middle-byte), which suggests the parses uses BOM to detect encoding. Not sure how/if that applies to a *response* stream... Also, I noticed you are using Lucee, while I am using ACF (*might* behave a little differently). I am not that familiar with jasperreports. Maybe later I can dig up an example to test the "fill" method. Try and reproduce the error. — Leigh, Sep 09 '15 at 13:12

score 1 · Accepted Answer · answered Sep 09 '15 at 13:43

With help of Leigh, I've solved the problem. The getInputStream() function did not use UTF-8. Even when I set the default file encoding to UTF-8, it didn't work.

To ensure, that the report generator used UTF-8, I've re-created the Input Stream and applied the encoding in the process.

jsonUrl = getParam('JSON_URL', ReportParamArray);

// Get the Input Stream
inputStream = CreateObject("java", "java.net.URL").init(jsonUrl).openConnection().getInputStream();

// Convert the stream to a byte array
bytes = CreateObject("java", "org.apache.commons.io.IOUtils").toByteArray(inputStream);

// Convert bytes to string
str = CreateObject("java", "java.lang.String").init(bytes);

// Create a new UTF-8 input stream
inputStreamUTF8 = CreateObject("java", "java.io.ByteArrayInputStream").init(str.getBytes("UTF-8"));

// Fill the report with our new input stream
jasperPrint = jasFillManager.fillReport(jasReport, parameters,
    CreateObject("java", "net.sf.jasperreports.engine.data.JsonDataSource").init(inputStreamUTF8));

This might not be the perfect solution in terms of performance, but it works and I escaped the encoding hell.

Is ColdFusion not capable of returning a valid UTF-8 JSON object?

1 Answers1