13

I'm trying to send an object like this to my REST API(built with asp net core)

{
    "firstName":"tersü",
    "lastName":"asda"
}

And this is how the headers form SoapUI look:

 Accept-Encoding: gzip,deflate
Content-Type: application/json:charset=UTF-16
Host: localhost:4004
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1.1 (java 1.5)

However, my actionContext.ModelState is always invalid because it can not work with the umlaute. The exception is the following:

Unable to translate bytes [FC] at index 35 from specified code page to Unicode

If it's any help, the method signature looks like this: 

[ValidateUserData]
public async Task<IActionResult> Update(string userId, [FromBody] UpdateUserRequest updateRequest)

Basically the code never goes over 

if (!actionContext.ModelState.IsValid)
{
    actionContext.Result = new BadRequestObjectResult(actionContext.ModelState);
}

inside the [ValidateUserData] attribute

What am I missing here?

DVM
  • 1,229
  • 3
  • 16
  • 22

4 Answers4

14

You are sending your string encoded in utf-16, but telling (in the Content-Type header's charset) it is utf-8.

The bytes for tersü in utf-8 are:

74,65,72,73,C3,BC

However tersü (in utf-16) contains the bytes (notice the FC there):

74,0,65,0,72,0,73,0,FC,0

(Check it in this fiddle)

So it just can't understand it. So either convert your string to utf-8 in your client before sending it, or set the Content-Type charset to utf-16 .

Jcl
  • 27,696
  • 5
  • 61
  • 92
  • I'm sorry, that was a mistake on my part. The headers i pasted are the response header, not the request ones. i've updated with request headers – DVM Aug 22 '16 at 11:43
2

Although the Content-Type is charset UTF-8 the received byte code FC denotes the extended ASCII character 252 which represents the umlaut "ü".

In a UTF-8 encoding the umlaut "ü" consists of two bytes. So there is a mismatch between the given encoding header and the transmitted data. So you have to check the code which generates the request.

Ralf Bönning
  • 14,515
  • 5
  • 49
  • 67
  • The fact that the index is 35, makes it much more bearable that the content being sent is actually utf-16 (not extended ascii)... and actually, extended ASCII's byte 252 (FC) is ³, not ü *(ü is 129 [hex 81])* – Jcl Aug 22 '16 at 10:19
  • 1
    @Jcl: I took the information about the FC hex code to character mapping from https://en.wikipedia.org/wiki/Extended_ASCII#/media/File:Table_ascii_extended.png and ftp://ftp.unicode.org/Public/MAPPINGS/ISO8859/8859-1.TXT. But where is the information what bytes are going over the wire in the environment of the OP ? – Ralf Bönning Aug 22 '16 at 10:28
  • 1
    actually, extended ascii depends on the codepage, so it's possible that we both are right :-) About the information, I'm just making that assumption since the error says at index 35... depends a bit on newlines and spaces that might be in the question and not in the actual data, but I'd say index 35 sounds like right for utf-16 and the presented data. I could definitely be wrong, but sounds like a safe assumption (plus: in .NET strings are encoded in utf-16 by default, and the OP is not mentioning anything else) – Jcl Aug 22 '16 at 10:49
  • @Jcl ok I see. Thanks for sharing your thoughts and assumptions. – Ralf Bönning Aug 22 '16 at 10:57
0

Not a direct solution to the question asked - but a solution for a related problem:

I was having this error message when downgrading a Winforms application from Net.4.0 to Net.2.0 (because of 3rd party form control issues)

Every instance of an ImageList object that was specified in the forms designer was affected.

I cut all such instances out of the RESX file and out of the Designer code, made them private objects, and did the required initialization in the Form_Load event.

No problems since then in Net.2.0 or Net 4.0 configurations.

Mike Pettigrew
  • 156
  • 1
  • 8
-1

I was having this error when working with SSRS on a .rdl file. The solution was that I needed to save the file as an unsigned UTF-8 file rather than a signed UTF-8 file. Here is how you do that in Visual Studio:

  1. Open your XML code by going to View -> Code, or pressing F7
  2. While viewing your XML code page, select File -> Advanced Save Options
  3. In the resulting pop-up windows select, "Unicode (UTF-8 without signature)..."
  4. Click OK

That fixed the problem for me, I hope it helps!

Joey C
  • 59
  • 2
  • 12