TIdHTTPProxyServer raising "Unknown Protocol" RSHTTPUnknownProtocol exception

Question

I'm recoding an old Delphi XE program using Delphi 10.3 Rio. It uses the TIdHTTPProxyServer Indy component listening on 127.0.0.1:80.

  with IdHTTPProxyServer.Bindings.Add do begin

    IP := '127.0.0.1';
    Port := 80;

  end;

  IdHTTPProxyServer.Active := True;

For testing, I added 127.0.0.1 localtest123.com and 127.0.0.1 www.localtest123.com to the hosts file and disabled the DNS cache service. Then in multiple browers I requested http://localtest123.com/ and http://www.localtest123.com/. With OutputDebugString() I can see the connections accepted but then raises an "Unknown Protocol" error.

I debugged the exception in the TIdHTTPProxyServer.CommandPassThrough procedure in IdHTTPProxyServer.pas. It seems LURI.Protocol is an empty string which is why the RSHTTPUnknownProtocol is raised.

  LContext := TIdHTTPProxyServerContext(ASender.Context);
  LContext.FCommand := ASender.CommandHandler.Command; //<-'GET'
  LContext.FTarget := ASender.Params.Strings[0]; //<-'/'

  LContext.FOutboundClient := TIdTCPClient.Create(nil);
  try
    LURI := TIdURI.Create(LContext.Target); //<-'/'
    try
      TIdTCPClient(LContext.FOutboundClient).Host := LURI.Host; //<-''

      if LURI.Port <> '' then begin //<-''
        TIdTCPClient(LContext.FOutboundClient).Port := IndyStrToInt(LURI.Port, 80);
      end
      else if TextIsSame(LURI.Protocol, 'http') then begin //<-''    {do not localize}
        TIdTCPClient(LContext.FOutboundClient).Port := IdPORT_HTTP;
      end
      else if TextIsSame(LURI.Protocol, 'https') then begin //<-'' {do not localize}
        TIdTCPClient(LContext.FOutboundClient).Port := IdPORT_https;
      end else begin
        raise EIdException.Create(RSHTTPUnknownProtocol);
      end;

I'm probably missing something but TIdHTTPProxyServer just works without much code so I have to ask for help on this exception. Thanks in advance!

Answer 1

You can't just redirect the domains in your HOSTS file and expect things to magically work. That is not how proxying works.

You must explicitly configure web browsers to make HTTP requests through an HTTP proxy so that they format the proper requests that a proxy would understand. Sending an HTTP request directly to a target web server is handled differently than sending the same HTTP request through a proxy.

You are getting the exception because the browser requests are not targeting your proxy properly.

For example, when a browser sends an HTTP GET request directly to a target web server, it connects directly to that server and then sends a request that looks something like this:

GET /path HTTP/1.1
Host: server.com

But, when it sends the same request through an HTTP proxy, it connects to the proxy and sends a request that looks more like this instead:

GET http://server.com/path HTTP/1.1

That extra path information in the GET line is missing in your browser requests, because you do not have your browsers configured for proxying, thus the exception when TIdHTTPProxyServer is trying to determine the info it needs to make a connection to the target web server and forward the current request to it.

This is fundamentally to how HTTP works, and how TIdHTTPProxyServer is designed to work.

Things are a bit more complicated when HTTPS is involved, but I'm leaving that detail out for now, as it is not relevant to your question about the exception.

UPDATE: in comments, you say:

In the XE version it never raised an exception when checking for the protocol which would still work today because I manually set the host and port in DoHTTPBeforeCommand.

In that old version, there was no exception raised, because TIdHTTPProxyServer did not check the protocol yet to differentiate between HTTP and HTTPS. You were able to manually fill in missing info when a request was received that was not specifically targeting your proxy. That is why things worked for you before.

In a later version, TIdHTTPProxyServer was updated to differentiate between HTTP and HTTPS when no port is explicitly specified in the request, so a default port is set based on protocol requested. That check happens before DoHTTPBeforeCommand() is called.

To get the old behavior back, you would have to alter TIdHTTPProxyServer's source code to delay the raising of the exception until after DoHTTPBeforeCommand() returns, so you have a chance to fill in missing values again.

If you file a feature request for that, I might consider adding it to Indy's official code.