Why it returns 403 when TIdHTTPProxyServer tries to forward the request to most of proxy servers?

Question

the question is extended from the question which I asked before.

Currently I can let my proxy server with TIdHTTPProxyServer forward the request to another proxy server which also using TIdHTTPProxyServer. But when I tried to let it forward the request to other proxy servers from the free proxy servers list on web. Most servers which can be used through IE proxy setting return 403 error message to my proxy server. In fact I have tried about 20 valid proxy servers, but only two can accept the requests from my proxy server. I can not figure out why it happens.

Below is the code I use in HTTPBeforeCommand of TIdHTTPProxyServer.

    TIdIOHandlerStack* tempIO = new TIdIOHandlerStack(AContext->OutboundClient);

    TIdConnectThroughHttpProxy* tempProxy = new TIdConnectThroughHttpProxy(AContext->OutboundClient);
    tempProxy->Enabled = true;
    tempProxy->Host = ProxyServerAddress;
    tempProxy->Port = ProxyServerPort ;
    tempIO->TransparentProxy  =  tempProxy;
    AContext->OutboundClient->IOHandler =  tempIO;

After monitoring the behaviors of TIdHTTPProxyServer using wireshark, I found TIdHTTPProxyServer always send a CONNECT request to other proxy servers at first berfore foward the real requests(the browser does not do that). And then receive 403 response for most proxy servers. But still do not know how to make it works.

Updated on 2012/08/07

Hi, I am not really familiar with those HTTP stuffs, so I just record what I saw in wireshark here. It seems IE uses GET/POST commands for HTTP requests and CONNECT command for HTTPS requests. And most proxy servers block Connect commands when they are not HTTPS request (For example, CONNECT www.google.com.tw:80 HTTP/1.0). That is why TIdConnectThroughHttpProxy always does not work.

Below is my workaround, I made a little bit changes in IdHTTPProxyServer.pas. Hope it is useful for someone else who meets the same problems.

For CONNECT commands, still use TIdConnectThroughHttpProxy

Within TIdHTTPProxyServer.CommandCONNECT

if UseProxy = True then
begin
     tempIO := TIdIOHandlerStack.Create(LContext.FOutboundClient);
     tempProxy := TIdConnectThroughHttpProxy.Create(LContext.FOutboundClient);
     tempProxy.Enabled := True;
       tempProxy.Host := UseProxyAddr;
       tempProxy.Port := UseProxyPort ;
       tempIO.TransparentProxy  :=  tempProxy;
       LContext.FOutboundClient.IOHandler :=  tempIO;
end;

For GET/POST commands, I should directly send GET www.google.com.tw:80 HTTP/1.0 to other proxy servers instead of sending CONNECT www.google.com.tw:80 HTTP/1.0 at first.

Within TIdHTTPProxyServer.CommandPassThrough

  if UseProxy = True then
  begin
     TIdTCPClient(LContext.FOutboundClient).Host := UseProxyAddr;
     TIdTCPClient(LContext.FOutboundClient).Port := UseProxyPort;
  end else
  begin
     TIdTCPClient(LContext.FOutboundClient).Host := LURI.Host;
     TIdTCPClient(LContext.FOutboundClient).Port := IndyStrToInt(LURI.Port, 80);
  end;

Also within TIdHTTPProxyServer.CommandPassThrough, I should let header Proxy-Connection = close

LContext.Connection.IOHandler.Capture(LContext.Headers, '', False);      
LContext.Headers.Values['Proxy-Connection'] := 'close';

Finally within TIdHTTPProxyServer.TransferData

if AContext.TransferSource = tsClient then 
begin

  if UseProxy = True then
  begin
     ADest.IOHandler.WriteLn(AContext.Command + ' ' + AContext.Target + ' HTTP/1.0'); 
  end else
  begin
      ADest.IOHandler.WriteLn(AContext.Command + ' ' + AContext.Document + ' HTTP/1.0'); 
  end;
end;

It is my first time to implement delphi, and hope there are better solutions.

Indy comes with full source code. So you could remove the CONNECT part and it should work if everything else is correct. — mjn, Aug 06 '12 at 10:30
The `CONNECT` is coming from `TIdConnectThroughHttpProxy`, not from `TIdHTTPProxyServer`, so the only way to avoid the `CONNECT` would be to stop using `TIdConnectThroughHttpProxy`. How does IE connect to those other proxies without using `CONNECT`? The only possibilities are either IE is sending a full URL on each `GET`/`POST` request instead of using `CONNECT`, or they are not HTTP proxies to begin with, maybe SOCKS proxies instead? — Remy Lebeau, Aug 06 '12 at 17:01
Right guess! IE sends a full URL on each GET/POST request to its proxy server like `GET http://www.google.com.tw/ HTTP/1.1`. I have some updates in my original question. Thanks! — Willy, Aug 07 '12 at 02:42

score 1 · Accepted Answer · edited May 23 '17 at 12:04

1

I believe you should not use pin-holing - http://www.indyproject.org/docsite/html/TIdConnectThroughHttpProxy.html

CONNECT command is not how WWW works. No browser use it. It is how non-WWW programs try to break-through the firewalls and open direct access to all areas of internet beyond WWW.

Don't use "transparent proxy" classes. Use regular HTTP proxy, like in How to download a file over HTTPS using Indy 10 and OpenSSL?

BTW, there is no such event handler as u name "HTTPBeforeCommand" http://www.indyproject.org/docsite/html/!!MEMBEROVERVIEW_TIdHTTPProxyServer.html

edited May 23 '17 at 12:04

Community

1
1

answered Aug 06 '12 at 09:07

Arioch 'The

15,799
35
62

Thanks for reply! You gave me the way to modify. I don't know why the document doesn't refer `HTTPBeforeCommand` event, but there is really a such one for TIdHTTPPRroxyServer in indy 10. And I summarize my comment in my last edit. – Willy Aug 07 '12 at 02:35

Why it returns 403 when TIdHTTPProxyServer tries to forward the request to most of proxy servers?

1 Answers1