jsonSerializer.Deserialize not working properly for unicode characters in c#

Question

I want to deserialise a JSON object containing unicode data to string array. While the characters inside the JSON object is english, it works fine. But when I use chinese it won't.

JavaScriptSerializer jsonSerializer = new JavaScriptSerializer();
string[] SampleText = jsonSerializer.Deserialize<string[]>(HttpContext.GetGlobalResourceObject("Resource", "SampleText").ToString());

When I run this in Immediate window of visual studio, the out comes like this for English

jsonSerializer.Deserialize<string[]>(HttpContext.GetGlobalResourceObject("Resource", "SampleText").ToString());
{string[3]}        
    [0]: "Size"
    [1]: "Name"
    [2]: "Type"

But for chinese an exception occurs

base {System.SystemException}: {"Invalid JSON primitive: ."}
Message: "Invalid JSON primitive: ."
ParamName: null

The resourse file value for english and chinese

["Size","Name","Type"]
[“大小”，“姓名”，“類型”]

Panagiotis Kanavos · Accepted Answer · 2014-08-20T12:22:35.340

1

EDIT: I just noticed that the japanese text is surrounded by smart quotes “ ” instead of actual quotes ". Replace the smart quotes with simple quotes. No language that I know of uses smart quotes as text delimiters, they are treated as content. Your text also uses a non-comma character ， (hex ff0c) instead of a comma.

Strings in .NET are Unicode. It isn't possible to create a non-Unicode string.

Problems occur only when you try to read non-Unicode content (like a file encoded in a specific codepage) as if it were a Unicode file. The OS (and .NET) will use the system locale to read non-unicode files which can result in garbled data. The solution is to either save files with Unicode encoding, or explicitly specify the file's encoding if it's different than the system locale.

In your case, the resource file is most likely not saved as a Unicode (or UTF8) file. Previous versions of Visual Studio saved files (including web pages) using the system's locale by default which resulted in some pretty interesting problems for non-US developers.

Examine the string returned by HttpContext.GetGlobalResourceObject("Resource", "SampleText").ToString(). The content will probably be garbled, which means the original string was not saved as Unicode.

The solution may be as simple as converting the resource file to Unicode.

edited Aug 20 '14 at 12:22

answered Aug 20 '14 at 11:45

Panagiotis Kanavos

120,703
13
188
236

Are those smart-quotes around the Japanese values? – Panagiotis Kanavos Aug 20 '14 at 12:07
I just [ctrl+c] and [ctrl+v] this. But dont know for two different languages how the same statement adds escape sequence only in english. – Arvin Aug 20 '14 at 12:09
This has nothing to do with languages. If there was an encoding problem the result would be garbled. It looks like you entered smart quotes instead of simple quotes when creating the resource. Only simple quotes can be used as string delimiters in Json (and all programming languages) – Panagiotis Kanavos Aug 20 '14 at 12:11
yes I useddouble quotes. But it is working fine for english. how ? – Arvin Aug 20 '14 at 12:15
I repeat, it has nothing to do with the language, or the text would be garbled. Apart from the smart quotes, your text also uses a **NON** comma character `，` as a separator. That's a **single** character in there and it isn't a comma – Panagiotis Kanavos Aug 20 '14 at 12:22
Yes you were right, the character actually looked similar to double quote but it was some other character.. Thanks – Arvin Aug 20 '14 at 13:15