JSON contains unicode characters

Question

My Java - Jersey framework REST API makes a call to another service which returns the following JSON response. I have logged the response from the child service in my logs, and I can see that the value of ErrorMessage contains a Unicode value like \u2019 instead of a single quote (').

{
  "id": "SAMPLE",
  "version": 1,
  "status": {
      "lastReceivedError": {
          "ErrorDateTime": 1576588715,
          "ErrorCode": "TEST3200",
          "ErrorMessage": "We\u2019re sorry, the content is not available."
      }
  }
}

I have to map these values into my model and return as a JSON as well. I used GSON to convert the above JSON string into an object. And mapped the values from that object into my response object. My final outgoing JSON response is like below, wherein the single quote is appearing as question mark (?).

{
    "MyResponse": {
        "success": {
            "lastReceivedError": {
                "errorDateTime": "2019-12-17T13:18:35Z",
                "errorCode": "TEST3200",
                "errorMessage": "We?re sorry, the content is not available."
            }
        }
    }
}

I believe there is something around encoding characters, but I am unable to fix the issue.

“My final outgoing JSON response is like below…” How do you know that? How are you examining it? If you print it in a Windows command window, characters outside the ISO 8859-x range are likely to appear as question marks. — VGR, Jan 20 '20 at 22:31

score 1 · Answer 1 · answered Jan 21 '20 at 07:46

TL;DR
Seeing is not believing. It depends on the encoding in your environment.

Code snippet
Following code snippet shows to deserialize the JSON string (part of original response).
If the encoding of your environment is UTF-8, then Gson will convert it correctly without specifying encoding.
And if you already knew the original string was encoded with UTF-8, you will get different results if you view it with UTF-8 and ISO-8859-1.

String jsonStr = "{\"ErrorMessage\": \"We\u2019re sorry, the content is not available.\"}";
Gson gson = new Gson();
JsonObject data = gson.fromJson(jsonStr, JsonObject.class);

System.out.println(data.toString());
System.out.println(new String(jsonStr.getBytes("UTF-8"), "UTF-8"));
System.out.println(new String(jsonStr.getBytes("ISO-8859-1"), "UTF-8"));

Console output

{"ErrorMessage":"We’re sorry, the content is not available."}
{"ErrorMessage": "We’re sorry, the content is not available."}
{"ErrorMessage": "We?re sorry, the content is not available."}

JSON contains unicode characters

1 Answers1