-1

Put a TIdHTTP component and a TIdSSLIOHandlerSocketOpenSSL component on a VCL form.

Do not assign anything to IdHTTP1.IOHandler property.

Then get the HTML from a HTTP URL, for example:

sHTML := IdHTTP1.Get('http://www.pixar.com');

• This is very fast and works well.

Then assign IdSSLIOHandlerSocketOpenSSL1 to IdHTTP1.IOHandler property.

Then put ssleay32.dll and libeay32.dll in the application directory.

Then get the HTML from a HTTPS URL, for example:

sHTML := IdHTTP1.Get('https://www.youtube.com/watch?v=CB1Pukr0nFQ');

• This is very slow and takes almost 4 (!) seconds to get the HTML.

So why this is so slow with the HTTPS URL and which is the fastest method to get the HTML both from HTTP and HTTPS URLs?

user1580348
  • 5,721
  • 4
  • 43
  • 105

1 Answers1

2

Encrypting and decrypting the HTML content has more overhead than transferring unencrypted content.

There is also overhead in establishing a secure session with the server before the content can then be requested and transferred (request and response are both encrypted).

Also, at the time when the HTTPS request is being made, if OpenSSL has not been loaded yet, there is overhead in loading and initializing the OpenSSL libraries.

Also, you are calling the version of TIdHTTP.Get() that returns a String, so all of the downloaded data has to be buffered in memory and then charset-decoded to UTF-16. The larger the HTML is, the more decoding that has to be done.

All of that overhead adds up.

Some things you can do to reduce some of the overhead are:

  1. pre-load OpenSSL beforehand (Indy has a LoadOpenSSLLibrary() function).

  2. make sure you are re-using HTTP connections to the same server, by using the same TIdHTTP/TIdSSLIOHandlerSocketOpenSSL objects, and enabling HTTP keep-alives by setting the TIdHTTP.Request.Connection property to keep-alive and/or setting the TIdHTTP.ProtocolVersion property to pv1_1 (keep-alive is the default behavior in HTTP 1.1, but not in 1.0).

  3. depending on what you are going to do with the downloaded HTML, you might consider downloading it into a TStream instead of a String. That way, TIdHTTP.Get() will skip the caching and charset decoding steps, whatever is downloaded will be saved as-is directly to your provided TStream. If you don't actually need the HTML at all, you can pass a nil TStream to tell Get() not to cache/process the downloaded data at all.

Also, check to see if sending a request to YouTube is actually generating only one request/response, and not maybe the server using HTTP redirects that cause additional requests to be sent, thus taking more time to reach the final content (see if the TIdHTTP.OnRedirect event is being triggered or not). Requesting Pixar's homepage is less likely to do that.

Remy Lebeau
  • 555,201
  • 31
  • 458
  • 770
  • 1
    So how does the browser do it so fast? When I open a Youtube in the browser, I get the page instantaneously. – user1580348 Dec 21 '15 at 09:14
  • Different browsers use different SSL/TLS implementations and do different things. Also, browsers also have a lot more processing power behind them. You are comparing apples to oranges here. In any case, I tested the URLs you provided using the latest SVN snapshot of Indy, and it takes `TIdHTTP.Get()` only 1.4 seconds to request, download, and decode the YouTube data. And that is taking into account all of the overhead I mentioned. – Remy Lebeau Dec 21 '15 at 21:45
  • I also noticed different response time among different open ssl library + Delphi version. Perhaps the library code is not perfectly optimized. – Giorgio Calzolato Jun 02 '18 at 16:21