Delphi idhttp broken characters

Question

I'm working on Delphi XE5 and Getting XML from server by using IDHTTP. Getting XML works fine but there are some broken character. The character is '•' (Bullet point). others are fine but the bullet point is broken.

I created IDHTTP like below:

idhttps := TIdHTTP.Create();
idhttps.IOHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil);
idhttps.IOHandler.DefStringEncoding := IndyTextEncoding(TEncoding.UTF8);
idhttps.HandleRedirects := True;
idhttps.ConnectTimeout := 5000;
idhttps.Request.USERNAME := 'USERNAME';
idhttps.Request.PASSWORD := 'PASSWORD';
idhttps.Request.BasicAuthentication := True;
idhttps.Request.Accept := 'text/xml';

And then getting xml like below:

SS := TStringStream.Create('', TEncoding.UTF8);

try
  self.GetIdHTTPForLexicomp.Get(URL, SS);
  XMLDoc := TXMLDocument.Create(nil);
  XMLDoc.LoadFromStream(SS, TXMLEncodingType.xetUTF_8Like);
finally
  SS.Free;
end;

In XML bullet point is displayed like below:

? Anaphylaxis/hypersensitivity: May cause hypersensitivity reactions,

XML header is below:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

What should i check?

Update: I added XML snippet. It needs an XSL file for style but in this case, I suppose that is not a problem. '?' is broken character.

<?xml version="1.0" standalone="yes"?>
<ns2:monogragh>
  <monograghFields>
    <field fieldId="234837" fieldTypeCode="war" created="2005-04-07T17:28:33Z" modified="2014-10-02T11:32:57Z" sectionId="0">
      <fieldName>Warnings/Precautions</fieldName>
      <content>
        <div id="war" class="block">
          <p style="text-indent:-2em;margin-left:2em;text-align:justify;">
            <b>
              <i>Concerns related to adverse effects:</i>
            </b>
          </p>
          <p style="text-indent:-2em;margin-left:4em;text-align:justify;">
            ? Anaphylaxis/hypersensitivity: May cause hypersensitivity reactions, including anaphylaxis; use with caution in patients with anaphylactic disorders.
          </p>
        </div>
      </content>
    </field>
  </monograghFields>
</ns2:monogragh>

It looks like i gave mis-information. i attached captured xml snippet. the first one is the result getting from browser by using rest client tool and the last one is the result getting xml thru idhttp.

Answer 1

1. Do not set the IOHandler.DefStringEncoding property when using TIdHTTP. Let TIdHTTP handle encodings its own ways.

2. Using a TStream to receive the XML is the correct choice. However, using a TStringStream in particular is not a good choice, because it is bound to the TEncoding you specify in the constructor. If the XML is not encoded in the same charset that the TEncoding implements, the XML would not be decoded properly. Use a TMemoryStream or TBytesStream instead, to preserve the original XML bytes as-is.

3. XML is self-describing when it comes to its encoding. Do not tell TXMLDocument the encoding it should use, let the XML itself tell TXMLDocument which encoding to use.

Try this:

idhttps := TIdHTTP.Create();
idhttps.IOHandler := TIdSSLIOHandlerSocketOpenSSL.Create(idhttps);
idhttps.HandleRedirects := True;
idhttps.ConnectTimeout := 5000;
idhttps.Request.USERNAME := 'USERNAME';
idhttps.Request.PASSWORD := 'PASSWORD';
idhttps.Request.BasicAuthentication := True;
idhttps.Request.Accept := 'text/xml';

MS := TMemoryStream.Create;
try
  idhttps.Get(URL, MS);
  MS.Position := 0;
  XMLDoc := TXMLDocument.Create(nil); // XMLDoc must be IXMLDocument, or a memory leak occurs
  XMLDoc.LoadFromStream(MS);
finally
  MS.Free;
end;

Now, TXMLDocument should be parsing the raw bytes that the server actually sends, without any interpretation by TIdHTTP or the RTL beforehand.

If you are still having the same problem, then either the XML itself is not properly encoded to begin with, or you are not processing/displaying the XML correctly after it has been loaded into TXMLDocument. Neither of which you have shown yet, so we can only guess where your actual problem lies, outside of what I mentioned above.