0

I'm using Delphi 10.4 (with patches 1, 2 and 3) on Windows 10 64bit to build a VCL Web Client application.

I have an API running on RAD Server, and I can get information from it using the below VCL Webclient application with no trouble. However, when I try to insert data into the API resources through a POST method, I got the following error message from the TRESRequest.Execute() method call:

No mapping for the Unicode character exists in the target multi-byte code page

Though I got this error message, the JSON data is being sent to the API and is saved to the database correctly. It seems that at a certain point of the Execute() method, an error is generated after the data has been sent to the server.

I've already tried TEncoding.UTF8.GetBytes(), TEncoding.ASCII.GetBytes(), and Unicode, without success.

This is my code:

procedure TForm1.BTPostClick(Sender: TObject);
var
  strjson : string;
  jo      : TJSONObject;
begin
  strjson :=
    '{'                                 +
      '"CUST_ID": 1500,'                +
      '"CUST_NAME": "John Doe Max",'    +
      '"CUST_COUNTERTOTAL": "500",'     +
      '"CUST_DETAILS": "This is just a test"' +
    '}';    

  //    jo := TJSONObject.ParseJSONValue(TEncoding.Unicode.GetBytes(StrJson),0) as TJSONObject;
  //    jo := TJSONObject.ParseJSONValue(TEncoding.ASCII.GetBytes(StrJson),0) as TJSONObject;
  jo := TJSONObject.ParseJSONValue(TEncoding.UTF8.GetBytes(StrJson),0) as TJSONObject;    

  Memo1.Clear;
  Memo1.Lines.Add(jo.ToString);

  RestRequest1.ClearBody;
  RestRequest1.Method := rmPost;
  RestResponse1.ResetToDefaults;
  RestRequest1.AddBody(jo);
  RestRequest1.Execute;   // ==> error: No mapping for the Unicode character exists in the target multi-byte code page
end;

SERVER SIDE code:

procedure TMytestResource1.MytestPost(const AContext: TEndpointContext; const ARequest: TEndpointRequest; const AResponse: TEndpointResponse);
var
  s : string;
  i : integer;
  jo : TJSONObject;
begin
  jo := TJSONObject.Create;
  jo.AddPair('Response Line 1','Apple is maçã');
  jo.AddPair('Response Line 2','Foot is pé ');
  jo.AddPair('Response Line 3','There is lá');
  jo.AddPair('Exceção  Line 3',TJsonString.Create('Exception is exceção'));
  //s := jo.ToString;  at this point the characters has accents
  AResponse.Body.SetValue(jo, True);
end;

Configuration of REST components is as follows :

RestClient1.Accept          := 'application/json, text/plain; q=0.9, text/html;q=0.8,';
RestClient1.AcceptCharset   := 'utf-8;q=0.8';
RestClient1.AcceptEnconding := 'utf-8';
RestClient1.FallbackCharsetEncoding := 'utf-8';
RestClient1.ContentType    := 'application/json'
    
RestRequest1.Accept          := 'application/json, text/plain; q=0.9, text/html;q=0.8,'
RestRequest1.AcceptCharset   := 'utf-8;q=0.8';
    
RestResponse1.ContentEncoding := '';
RestResponse1.ContentType := '';

New Server Side Code (worked fine):

procedure TMytestResource1.CadastraUser(const AContext: TEndpointContext; const ARequest: TEndpointRequest; const AResponse: TEndpointResponse);
var
  s : string;
  i : integer;
  jo : TJSONObject;
  Strm : TStringStream;
begin
  jo := TJSONObject.Create;
  jo.AddPair('Response Line 1','Apple  is maçã');
  jo.AddPair('Response Line 2','Foot   is pé');
  jo.AddPair('Response Line 3','There  is lá');
  jo.AddPair('Response Line 4','Useful is útil');

  //AResponse.Headers.SetValue('Content-Type', 'application/json; charset=utf-8');  ==> This line does not make any effect on response

  try
    // Strm := TStringStream(jo.ToJSON, TEncoding.UTF8); ==> does not work.
    Strm := TStringStream.Create(jo.ToJSON, TEncoding.UTF8);
    // AResponse.Body.SetStream(Strm, 'application/json; charset=utf-8', true); ==> This line causes an error on VCL WebClient during RestRequest.Execute() method , error is "Invalid enconding name."

    AResponse.Body.SetStream(Strm, 'application/json', true); // this works fine.
  finally
    jo.Free;
  end;   
end;

So now, new questions arise :

  1. Why does this same program work fine in Delphi 10.3.1 Tokyo under Windows 7, but now in Delphi 10.4 Sydney (patches 1,2,3) on Windows 10 it requires UTF-8 encoding? Is it a Delphi 10.4 issue? Is it a Windows 10 issue?

  2. In my original server side code, I have many Endpoints sending responses using the command AResponse.Body.SetValue(myJSONObject, True); Should I replace all of them by the two new commands?

    Strm := TStringStream.Create(jo.ToJSON, TEncoding.UTF8);
    AResponse.Body.SetStream(Strm, 'application/json', true);
    

Code of REST.Client unit where the error is occuring :

Procedure TCustomRestRequest.Execute 

...
              if LMimeKind <> TMimeTypes.TKind.Binary then
              begin
                LEncoding := TEncoding.GetEncoding(FClient.FallbackCharsetEncoding);  //==> This line is returning nil, though FClient.FallbackCharsetEncoding is filled with 'utf-8'
                LContentIsString := True;
              end;
            end
            else
            begin
              // Even if no fallback, handle some obvious string types
              if LMimeKind = TMimeTypes.TKind.Text then
                LContentIsString := True;
            end;
          end;
          if LContentIsString then
            LContent := FClient.HTTPClient.Response.ContentAsString(LEncoding); // ==> this line is generating the error 
        finally
          LEncoding.Free;
        end;
...
JRG
  • 513
  • 9
  • 23
  • What does the server send in response to this request? The error is likely happening when decoding the response, not when preparing the request. Also FYI, `TJSONObject.ParseJSONValue()` has an overload that takes a `String` as input, so you don't need to convert `strJson` to `TBytes` at all. Also, I would suggest simply creating a fresh `TJSONObject` manually instead of using `ParseJSONValue()`, eg: `jo := TJSONObject.Create; jo.AddPair('CUST_ID', TJSONNumber.Create(1500)); jo.AddPair('CUST_NAME', 'John Doe Max'); ...` – Remy Lebeau Oct 13 '20 at 22:31
  • @RemyLebeau, this is the JSON response from the server { "Status " : "n�o tem erro", "Message": "Inclus�o conclu�da com sucesso!", "Record": "125 inclu�do.", "ID": "125"} exactly with those unreadable characters from Portuguese language accents. Regarding TJSONObject.ParseJSONValue(), I'm only choose this Parse method passing a TBytes just to apply UTF-8 conversion as an attempt to solve this unicode error message. It seems the issue is in the response, once RestRequest1.Execute really posted the data into the database. – JRG Oct 14 '20 at 05:51
  • @Remy, I did further tests and built an RAD Server Resource that just send a JSON response with and without accent words. In the VCL Client side , when receiving response with accent words it generates the error message "No mapping for the Unicode character exists... " , but when receiving only regular characters , it works fine ! My conclusion is that problem is on the RAD Server side when generating the JSON string on command AResponse.Body.SetValue(myJSONObject, True) . Do you have any clue how can I solve this ? Thks. – JRG Oct 14 '20 at 07:00
  • Please [edit] your question to show the actual server code that is creating the response, and the **complete** raw response data including HTTP headers. When dealing with character encodings, it is important to see everything you are doing to know if you are missing sonething, such as not setting a correct charset. – Remy Lebeau Oct 14 '20 at 14:30
  • @Remy, please see my server side code and the configuration of rest components.Thks. – JRG Oct 14 '20 at 15:40
  • You did not provide the *raw HTTP response* like I asked for, that is very important to look at. But in any case, `'utf-8'` is not a valid entry for `AcceptEncoding`, and I don't see you assigning a charset for the response. Try this: `AResponse.Headers.SetValue('Content-Type', 'application/json; charset=utf-8')`. Alternatively: `jo := ...; try Strm := TStreamStream(jo.ToJSON, TEncoding.UTF8); AResponse.Body.SetStream(Strm, 'application/json; charset=utf-8', true); finally jo.Free; end;` – Remy Lebeau Oct 14 '20 at 16:53
  • @Remy, sorry for not having provided the code as you asked, In fact I don't know how to get the raw response data. I've adjusted the code you proposed and finally it worked. I posted final code and commented parts that haven't work. I'm confused now about the reasons my code stopped working on Delphi 10.4 on Windows 10 (see questions Q1 and Q2) is it a Delphi issue or Windows 10 issue? Thanks ! – JRG Oct 15 '20 at 02:32
  • @Remy, any idea on why this program is presenting wrong characters on Delphi 10.4 on Windows 10 ? Why is it working only with TStringStream encoded in UFT-8? Thks. – JRG Oct 15 '20 at 13:32
  • I can't answer that without seeing RAD Server's source code for 10.3 and 10.4 (which I don't have), or seeing the raw HTTP response data. It is obviously a charset mismatch in how `Body.SetValue()` prepares the response vs how `Body.SetStream()` prepares it – Remy Lebeau Oct 15 '20 at 14:27
  • @Remy, once I have both installed, is there something I can provide you in order to you evaluate it? How can I get the HTTP raw data of the response ? – JRG Oct 15 '20 at 14:53
  • let's start simpler. Is the `TRESTResponse.Headers` being populated before the error is raised? If so, what do those headers look like? I'm mainly interested in seeing the raw `Content-Type` header and whether it specifies a `charset` or not. – Remy Lebeau Oct 15 '20 at 15:40
  • @Remy, ok, I tried this : on Client side, RestResponse1.Headers.text is empty immediately before call Restrequest1.Execute and after call it I could not check RestResponse1.Headers because it raised an exception with the above mentioned error. By debbuging the Restrequest1.Execute I found the error is raised on REST.Client unit at line 3091 on method procedure TCustomRESTRequest.Execute, I put the source code in the editor. I hope this can help. – JRG Oct 15 '20 at 19:39
  • "*after call it I could not check RestResponse1.Headers because it raised an exception with the above mentioned error*" - so? Just **catch the error** with a `try..except`, or **use the debugger** to view the `Headers` when the exception causes the debugger to pause the app. – Remy Lebeau Oct 15 '20 at 19:50
  • "*I put the source code in the editor*" - You did not show all of the relavent code. What does `LEncoding` get initialized with before `FallbackCharsetEncoding` is used? What condition causes `Execute()` to even look at `FallbackCharsetEncoding`? The lack of a `charset` in the response? What encoding does `ContentAsString()` use if passed a `nil` encoding? I don't have the REST source code to look at myself (I told you that earlier). But in any case, if `TEncoding.GetEncoding('utf-8')` is returning `nil` then that is a serious bug in the RTL, since UTF-8 is a standard encoding. – Remy Lebeau Oct 15 '20 at 19:58
  • @Remy, as you observed about TEncoding.GetEncoding('utf-8') returning nil, I also think this is a serious bug in this version, which not occur in the previous one. Instead of going in more debug sessions and spend my , and yours, precious time, I'll upgrade to the next release 10.4.1 . I'll let you know the result! Thanks a lot Remy ! – JRG Oct 18 '20 at 07:53
  • @RemyLebeau, I upgraded to 10.4.1 and no more problems! So I concluded this was a bug in 10.4 patches 1,2,3. Thanks , for your support ! – JRG Oct 22 '20 at 09:38

0 Answers0