Support for UTF-8 encoded string in TRestClient Delphi XE5

Question

I want to send tweets using the new TRest components in Delphi XE5. I am looking for a way to UTF8 encode my tweet which contains IS0-8859-1 characters. The code below works, but involves codepage conversion etc. Is the a better way? Anyone?

procedure TTwitterApi.Send(Tweet: string);
begin
  Reset;

  // Encode as UTF8 within (UTF-16 Delphi) string
  Tweet := EncodeAsUTF8(Tweet);

  FRestRequest.Resource := '1.1/statuses/update.json';
  FRestRequest.Method := rmPOST;
  FRestRequest.Params.AddItem('status', Tweet, pkGETorPOST);
  FRestRequest.Execute;
end;


function TTwitterApi.EncodeAsUTF8(UnicodeStr: string): string;
var
  UTF8Str: AnsiString;
  TempStr: RawByteString;
begin
  TempStr := UTF8Encode(UnicodeStr);
  SetLength(UTF8Str, Length(TempStr));
  Move(TempStr[1], UTF8Str[1], Length(UTF8Str));
  Result := UTF8Str;
end;

Remy Lebeau · Accepted Answer · 2016-08-19T19:26:06.773

Twitter's 1.1/statuses/update.json URL expects data to be encoded in application/x-www-form-urlencoded format, so you need to set the TRESTClient.ContentType property to ctAPPLICATION_X_WWW_FORM_URLENCODED (it is set to ctNone by default).

As for UTF-8, TRESTClient uses Indy internally, and Indy supports encoding outbound data using user-specified charsets, but it does not appear that Embarcadero added that feature to its TRESTClient interface (it does handle charsets in responses, though). I do not know why Embarcadero would omit such an important feature. It is not enough to just encode the string data as UTF-8 (which you are not doing correctly, BTW), but you also have to tell Twitter that the data has been UTF-8 encoded (via the charset attribute of the Content-Type REST header), and TRESTClient does not allow you to do that, as far as I can see. I don't know if TRESTClient sends REST requests with a default charset specified, but looking at its source, I don't think it does, but I have not tried it.

At the very least, you need to fix your EncodeAsUTF8() function. It does not produce a UnicodeString that holds UTF-8 encoded octets, like you think it does. It produces a UTF-8 encoded AnsiString and then converts that to a UTF-16 encoded UniodeString using the RTL's default Ansi codepage, so you are invoking a data conversion that loses the UTF-8 data. Try this instead:

function TTwitterApi.EncodeAsUTF8(UnicodeStr: string): string;
var
  UTF8Str: UTF8String;
  I: Integer;
begin
  UTF8Str := UTF8String(UnicodeStr);
  SetLength(Result, Length(UTF8Str));
  for I := 1 to Length(UTF8Str) do
    Result[I] := Char(Ord(UTF8Str[I]));
end;

That should allow TRESTClient to url-encode the correct UTF-8 data in its POST data, at least. But you still have to deal with the issue of the missing charset attribute in the Content-Type request header (unless Twitter defaults to UTF-8 when no charset is specified).

Now, with all of that said, if you find that working around the TRESTClient problems does not work out for you, then I would suggest switching to Indy's TIdHTTP component instead (which has a more accurate application/x-www-form-urlencoded implementation than TRESTClient is using), eg:

procedure TTwitterApi.Send(Tweet: string);
var
  Params: TStringList;
begin
  Reset;

  Params := TStringList.Create;
  try
    FParams.Add('status=' + Tweet);
    FIdHTTP.Request.ContentType := 'application/x-www-form-urlencoded';
    FIdHTTP.Request.Charset := 'utf-8';
    FIdHTTP.Post('https://api.twitter.com/1.1/statuses/update.json', Params, IndyTextEncoding_UTF8);
  finally
    Params.Free;
  end;
end;

I have looked at ContentType in the REST.Client module. With a POST the ContentType is taken from the request params. (Se line 2569: LContentType := ContentType;) If the contenttype is ctNone it actually ends up as APPLICATION_X_WWW_FORM_URLENCODED. So the ContentType appers to be correct. — user3012503, Nov 22 '13 at 13:45
As you say there is no way of setting UTF-8 as the charset, but since Twitter only accepts UTF-8 this works. — user3012503, Nov 22 '13 at 13:49
Thanks for the advice to use the TIdHttp component directly. The only disadvatange I can see is that I need to work out the OAuth calls — user3012503, Nov 22 '13 at 14:10
Indy does not natively support OAuth yet (it is on the todo list), but I have seen some third-party OAuth implementations using Indy floating around, including some that are Twitter-related. — Remy Lebeau, Nov 22 '13 at 16:12
The code of the function TTwitterApi.EncodeAsUTF8 is wrong. It do not compile, the TempStr var do no exist. I check it must be UTF8Str, isn't it ? — Rodrigo Farias Rezino, Aug 19 '16 at 13:34

score 0 · Answer 2 · answered Aug 13 '15 at 22:47

0

TRestRequest don't works with android, it causes a number of problems, specially with UTF8, that I could not solve, IdHttp Indy do works fine.

answered Aug 13 '15 at 22:47

Flávio Filho

475
4
14

score 0 · Answer 3 · answered Feb 06 '18 at 12:10

I've solved this issue with a different API provider (not Twitter) in the following way:

function EncodeAsUTF8(UnicodeStr: string): AnsiString; // <-- Note the Ansi
var
  UTF8Str: UTF8String;
  I: Integer;
begin
  UTF8Str := UTF8String(UnicodeStr);
  SetLength(Result, Length(UTF8Str));
  for I := 1 to Length(UTF8Str) do
    Result[I] := AnsiChar(Ord(UTF8Str[I])); // <-- Note the Ansi
end;

...

fRESTClient1 := TRESTClient.Create(nil);
fRESTClient1.Accept := 'application/json';
fRESTClient1.AcceptCharset := 'UTF-8';
fRESTClient1.AcceptEncoding := 'identity';
fRESTClient1.ContentType := 'application/x-www-form-urlencoded';

...

rrOrder := TRESTRequest.Create(nil);
rrOrder.Accept := 'application/json';
rrOrder.AcceptCharset := 'UTF-8';
rrOrder.Client := fRESTClient1; {}
rrOrder.Method := rmPOST;
rrOrder.Resource := 'xxxxxx';
rrOrder.Params.AddItem('', EncodeAsUTF8(aJson), pkREQUESTBODY, [poDoNotEncode]);

rrOrder.Execute;

Support for UTF-8 encoded string in TRestClient Delphi XE5

3 Answers3