POST with Indy + SSL + Proxy

Question

I'm trying to do a POST request through proxy using https. Code looks like:

  FHttp := TIdHttp.Create(nil);

  FHttp.ProxyParams.ProxyServer := Host;
  FHttp.ProxyParams.ProxyPort := Port;
  FHttp.ProxyParams.ProxyUsername := User;
  FHttp.ProxyParams.ProxyPassword := Password;

  FHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil);
  FHandler.SSLOptions.Method := sslvTLSv1_2;
  FHandler.PassThrough := true;
  FHttp.IOHandler := FHandler;
  FHttp.HandleRedirects := true;
  FHttp.Request.ContentType := 'application/x-www-form-urlencoded';
  FHttp.Request.Connection := 'keep-alive';
  FHttp.Request.ProxyConnection := 'keep-alive';
...

  FParams.Add('username=user');
  FParams.Add('password=pwd');
  FHttp.Post('https://my.service/login', FParams);

Proxy server is Squid.

Code generates error "Socket Error # 10054 Connection reset by peer."

Now, the interesting part comes:

1. If not using proxy at all (i.e. not setting FHttp.ProxyParams settings) - everything is OK.
2. If not setting any POST parameters (i.e. empty FParams), but still using proxy - everything is OK.
3. The most strange one: If I'm debugging the Indy code step by step (TIdCustomHTTP.DoRequest method) - everything is OK with above example (proxy settings + parameters).

POST parameters are not sent properly for some reason?

And why step 3 is happening?

Indy is up to date, just pulled from repository

UPDATE

After intercepting TIdHTTP calls (thanks Remy) there is a little bit more clarity. (failing log, working log).

Short version: when doing debug, Indy does 3 CONNECT + POST + DISCONNECT requests (because there are redirection on the service I believe) and it works.

When running test without debug - CONNECT + DISCONNECT + POST - and it fails obviously (i.e. POST is executed without CONNECT in front). See attached log files for details.

Answer 1

You have found some logic bugs in TIdHTTP that need to be fixed. I have opened a new ticket for that:

#315: Bugs in TIdHTTP proxy handling

---

Here is what I see happening in your "failing" scenario:

TIdHTTP connects to the proxy, sends a CONNECT request that successfully connects to my.service.com:443, then sends a POST request (using HTTP 1.0 rather than HTTP 1.1 ^a).

^{a) to send a POST request with HTTP 1.1, you have to set the TIdHTTP.ProtocolVersion property to pv1_1, AND enable the hoKeepOrigProtocol flag in the TIdHTTP.HTTPOptions property. Otherwise, TIdHTTP.Post() forces the ProtocolVersion to pv1_0.}

The HTTP server replies with a 302 Found response redirecting to a different URL, including a Keep-Alive header indicating the server will close the connection if a new request is not sent in the next 5 seconds.

When TIdHTTP is done processing the POST response, it knows it is going to re-send the same request to a new URL. On the next loop iteration, it sees that the target server is the same, and the proxy is still connected, and so the connection is not closed, and the code that would have sent a new CONNECT request is skipped.

Just before the POST request is sent, the Response.KeepAlive property is checked to know whether or not to close the socket connection anyway. The KeepAlive property getter sees the ProtocolVersion property is pv1_0 and that there is no Proxy-Connection: keep-alive header present in the response (even though there is a Connection: keep-alive header), so it returns False, and then the socket connection is closed.

TIdHTTP then re-connects to the proxy again, but does not send a new CONNECT request before sending the POST request. The proxy does not know what to do with the POST, so it fails the request with a 400 Bad Request response.

---

Here is what I see happening in your "working" scenario:

Everything is the same as above, up to the point where the 1st POST request is processed. Then there is a delay of roughly 16 seconds (likely since you are stepping through code) - more than the 5-second Keep-Alive delay allows - so the HTTP server closes its connection with the proxy, which then closes its connection to TIdHTTP.

By the time TIdHTTP is ready to send the 2nd POST request, it knows it has been disconnected from the proxy, so it re-connects to the proxy, sends a new CONNECT request, and then sends the POST request.

---

Until I can fix the bugs properly, try the following:

• enable the hoKeepOrigProtocol flag in the TIdHTTP.HTTPOptions property to allow TIdHTTP.Post() to use HTTP 1.1. That in itself may fix the issue with the connection being closed unnecessarily before sending the 2nd POST request to the redirected URL.

• if that doesn't solve the issue, try editing IdHTTP.pas yourself and recompile Indy, to update the TIdCustomHTTP.ConnectToHost() method to force a Disconnect() if the Response.KeepAlive property is False BEFORE the local LUseConnectVerb variable is set to not Connected in the case where ARequest.UseProxy is ctSSLProxy (and ctProxy, too). That way, the 2nd POST request will disconnect from the proxy and re-connect with a new CONNECT request.