Why is unicode encoded differently in these two scenarios?

Question

I have written this minimal example:

public static class X {
    private String x;

    public String getX() {
        return x;
    }

    public void setX(String x) {
        this.x = x;
    }

    public X(String x) {
        super();
        this.x = x;
    }

}
@RequestMapping(value = "/unicode.test1", method = RequestMethod.GET, produces = MediaType.APPLICATION_JSON_VALUE)
@ResponseBody
public String unicodeTest1() {
    return "{\"x\":\"X\u00ADX\"}";
}
@RequestMapping(value = "/unicode.test2", method = RequestMethod.GET, produces = MediaType.APPLICATION_JSON_VALUE)
@ResponseBody
public X unicodeTest2() {
    return new X("X\u00ADX");
}

Why does each of the two endpoints return a different result?

More importantly, which is of the two results is strictly correct as of 2019 standards and best practices?

HEADERS

C:\>curl -i "http://localhost/rets_api/unicode.test1"
HTTP/1.1 200
Content-Type: application/json;charset=ISO-8859-1
Content-Length: 11
Date: Mon, 18 Nov 2019 06:24:01 GMT

{"x":"X¡X"}
C:\>curl -i "http://localhost/rets_api/unicode.test2"
HTTP/1.1 200
Content-Type: application/json;charset=UTF-8
Transfer-Encoding: chunked
Date: Mon, 18 Nov 2019 06:24:05 GMT

{"x":"X┬¡X"}

It looks like your `unicodeTest1()` isn't being encoded _at all_, because otherwise you'd get a JSON string value instead of the JSON object you have. (Also, for encoding questions post a hex dump, not a terminal screenshot with interpreted characters.) — chrylis -cautiouslyoptimistic-, Nov 18 '19 at 05:59
Looks like I needed to use `produces = MediaType.APPLICATION_JSON_UTF8_VALUE` in the first variant to make it the same as the 2nd one. — Alex R, Nov 18 '19 at 06:28

score 2 · Accepted Answer · edited Oct 07 '21 at 11:04

In the first case Spring uses your default charset (ISO-8859-1), while in the second, when Spring is responsible for JSON serialization, UTF-8.

From the RFC:

JSON text SHALL be encoded in UTF-8, UTF-16, or UTF-32. The default
encoding is UTF-8, and JSON texts that are encoded in UTF-8 are
interoperable in the sense that they will be read successfully by the maximum number of implementations; there are many implementations
that cannot successfully read texts in other encodings (such as
UTF-16 and UTF-32).

You can explicitly specify UTF-8 using produces = MediaType.APPLICATION_JSON_UTF8_VALUE or by configuring your OS.

Why is unicode encoded differently in these two scenarios?

1 Answers1