TIdHTTP sends HTTPS request as HTTP over proxy

Question

I've got an issue with implementing SSL Certificate Pinning with TIdHTTP.

So, here are the steps:

1. Drop TIdHTTP, TIdSSLIOHandlerSocketOpenSSL and TIdCompressorZLib on the form.
2. Assign TIdSSLIOHandlerSocketOpenSSL and TIdCompressorZLib to IOHandler and Compressor properties of TIdHTTP.

3. Setup TIdSSLIOHandlerSocketOpenSSL:

   Port = 0
   DefaultPort = 0
   SSLOptions.Method = sslvTLSv1_2
   SSLOptions.SSLVersions = [sslvTLSv1_2]
   SSLOptions.Mode = sslmClient
   SSLOptions.VerifyMode = [sslvrfPeer]
   SSLOptions.VerifyDepth = 0
   OnVerifyPeer = SSLIOHandlerVerifyPeer
   

4. Code for SSLIOHandlerVerifyPeer:

   function TForm2.SSLIOHandlerVerifyPeer(Certificate: TIdX509; AOk: Boolean; ADepth, AError: Integer): Boolean;
   const
     LCGoogleCert = '98:1D:34:C4:F8:4A:F2:B7:C7:AB:77:AD:51:1C:51:4C:AD:76:ED:0D:0E:FA:C9:63:68:AF:28:69:94:60:BF:7A';
   begin
     Result := Certificate.Fingerprints.SHA256AsString.Equals(LCGoogleCert);
   end;
   

5. Drop a button on a form:

   procedure TForm2.Button1Click(Sender: TObject);
   const
     LCGoogleURL = 'https://www.google.com/';
   var
     s: UnicodeString;
   begin
     s := HTTPSender.Get(LCGoogleURL);
   end;
   

6. Install Fiddler

7. In Fiddler: Tools - Options - check "Capture HTTPS Connects" and uncheck "Decrypt HTTPS traffic". Generate and install certificate to system.

8. Set Fiddler's proxy address and port to TIdHTTP.

9. Run program and click the button. First click - you get the exception about the incorrect certificate. But if you click the second time - you won't get any exceptions, but you'll get the full response and you'll see the unencrypted traffic in Fiddler as if you sent the request via HTTP and not HTTPS.

You can see the result of the first and the second request on the picture below. So is this a bug in Indy component, or I'm trying to implement SSL Pinning incorrectly?

Answer 1

On the first call to TIdHTTP.Get(), the OnVerifyPeer event sees Fiddler's SSL/TLS certificate instead of Google's, so it rejects the certificate, and the underlying socket connection ends up being closed.

However, there is unread data left behind in the IOHandler's InputBuffer (approx 162 bytes of encrypted data). By design, the TIdIOHandlerStack.Connected() method returns True as long as there is unread data available to satisfy read operations, even if there is no physical socket connection.

So, what ends up happening during the second call to TIdHTTP.Get() is the following:

• TIdHTTP knows it is communicating over HTTPS through a proxy, and that it is making a new HTTP request to the same Google server through the same proxy as the previous call to TIdHTTP.Get(). So TIdHTTP checks Connected() to see if it is still connected to the proxy, and sees that Connected() is initially True, so it makes the decision to skip a new CONNECT request and proceed as if it is sending a new HTTP request over an existing proxy connection.

• However, since the underlying socket was disconnected, TIdHTTP has to make a new socket connection to Fiddler. While preparing for the new HTTP request, the InputBuffer gets cleared. Connected() is checked again, and is now False, so TIdHTTP makes a new socket connection to Fiddler. That new socket connection is initially unencrypted (the IOHandler's PassThrough is set to True) so a subsequent CONNECT would not be encrypted (but this section of code does not know that TIdHTTP had already decided to skip CONNECT).

• TIdHTTP proceeds to send the GET request to Fiddler, unencrypted.

• Fiddler caches its TLS tunnels for a period of time, and so it reuses the existing tunnel to Google, thus forwarding the unencrypted GET as-is to Google over a TLS connection, and then forwards the response unencrypted back to TIdHTTP.

So, ultimately, there are three issues at play here (I have opened a ticket in Indy's issue tracker):

• TIdHTTP is not clearing the InputBuffer when a failure occurs that requires the underlying socket to be closed. A simply fix for this is to make the TIdCustomHTTP.ConnectToHost() method clear the InputBuffer of any existing data before doing anything else. That way, it sees the connection is really gone before deciding what to do about CONNECT. I have now checked in this fix to Indy's SVN repository, and tested that it works in your scenario.

• TIdHTTP is making the decision to send or skip CONNECT too soon, before it knows what it is doing with the underlying socket. This will require some re-writing of TIdHTTP's internal logic, so it will be deferred to a later version of Indy.

• Fiddler is forwarding unencrypted data back and forth over a previously encrypted tunnel. Nothing can be done about that in TIdHTTP.