0

Is it possible to implement something like IdMappedPortTCP without connecting to a remote proxy server?

What I need is a) a way to edit every HTTP header (for example change the User-Agent for each request) the for every request sent from my computer without having to connect to a remote server. And b) If possible, I would also like to capture all the http traffic in delphi without the need of a third party application like proxifier.

What i have tried so far is:

a) IdMappedPortTCP and then binding to a remote proxy server, I then modify the AThread.NetData in each request in the IdMappedPortTCPExecute method.

b) using proxifier to capture all http traffic in the computer.

What I have tried so far is using Mapping with IdMappedPortTCP to a local proxy server (e.g. squid, delegate, fiddler, ccproxy), create my own proxy server (using indy 10) - All these worked great for HTTP connections but require installation of root certificate to modify HTTPS requests which is undesired. If its possible to implement any local proxy without having to install root certificates it would be awesome!

I have also tried to modify the TCP REDIRECTOR CODE but being fresh n all in programming, i haven't been successful. I figured i could change the 

procedure TForm1.IdTCPServer1Execute(AThread: TIdPeerThread);
var
  Cli: TIdTCPClient;
  Len: Cardinal;
  Data: string;
begin
  try
    Cli := nil;
    try
     { Create & Connect to Server }
     Cli := TIdTCPClient.Create(nil);
     Cli.Host := 'www.borland.com';
     Cli.Port := 80;
     { Connect to the remote server }
     Cli.Connect;
     ..............

Such that I would extract the host and port from request and then assign cli.host to that host and port dynamically for each request. I don't know how viable that is. Like, would it cause computer to hang because of connecting to too many remote host?

Update: with TIdMappedPortTCP, I used AThread.Connection.Capture(myheaders,''); so now I can assign my host to myheaders.Values['host'] and if AThread.Connection.ReadLn = 'CONNECT' I set port to 443 otherwise I set it as 80. Am I on the right track?

procedure TForm1.IdMappedPortTCP1Connect(AThread: TIdMappedPortThread);
var
  myheaders: TIdHeaderList;
  method : string;
begin
  myheaders:=TIdHeaderList.Create;
  try
    method:= AThread.Connection.ReadLn;
    Athread.Connection.Capture(myheaders);        
    if myheaders.Count<>0 then begin
      if Pos('CONNECT',method)<>0 then begin
       with TIdTCPClient(AThread.OutboundClient) do begin
         Host:=myheaders.Values['host'];
         Port:=443;
       end;
     end else begin
       with TIdTCPClient(AThread.OutboundClient) do begin
         Host:=myheaders.Values['host'];
         Port:=80;
       end;
     end;
     TIdMappedPortThread(AThread).NetData:= method + #13#10 + myheaders.Text + #13#10 + #13#10;
    end else begin
      TIdMappedPortThread(AThread).NetData:= method + #13#10 + #13#10;
      outs.Lines.Add(TIdMappedPortThread(AThread).NetData);
    end;
  finally
    myheaders.Free;
  end;
end;

I have put that code in the OnConnect event but it does not seem to be working. What have I done wrong?

Wanyiri
  • 1
  • 5
  • 1
    You have to act as a proxy for HTTP clients to connect to (and each client would have to be configured accordingly), so they can send their HTTP traffic to you and then you can modify it and pass it on to the real server as needed (you don't need to connect to another proxy unless your network requires it). `TIdMappedPortTCP` is NOT good for implementing an HTTP proxy. Use `TIdTCPServer` directly instead and implement the actual HTTP protocol. But you cannot implement a modifying HTTP proxy without a certificate since you have to act as a man-in-the-middle attacker to decrypt HTTPS data. – Remy Lebeau Sep 16 '15 at 01:07
  • Without acting as a proxy, you are left to capture raw packets off the Ethernet adapter before they go out the wire, but then you cannot process encrypted HTTPS data, and you might not be able to modify unencrypted HTTP data, depending on which intercept/capture driver you decide to install to access the raw packets. – Remy Lebeau Sep 16 '15 at 01:07
  • Thanks @RemyLebeau for the reply. While using TIdMappedPortTCP, I can modify even the https request headers without the need of a certificate. The problem is that it requires a remote proxy..or a local one (which requires certificate). Can u please explain further how to go about with TidTCPServer directly? I tried packet capture using WinPCap driver to do that but modifying packets at that level seemed not useful. Or maybe I was doing something wrong – Wanyiri Sep 17 '15 at 09:03
  • `TIdMappedPortTCP` would be acting as the proxy for clients, you DO NOT need to connect it to another proxy (unless your PC is on a machine that needs a proxy to reach outside servers). Simply have `TIdMappedPortTCP` connect to the server the client asks for. If you implement HTTP on top of `TIdMappedPortTCP`, you need your own certificate, since `TIdMappedPortTCP` will be performing its own SSL/TLS handshake with the client. I suggest you upgrade to Indy 10 and use its `TIdHTTPProxyServer` component instead of using `TIdMappedPortTCP`. You still need your own certificate, though. – Remy Lebeau Sep 17 '15 at 17:20
  • @RemyLebeau _ you DO NOT need to connect it to another proxy Simply have TIdMappedPortTCP connect to the server the client asks for_ That is exactly what i want, but how do i implement that. I use `IdMappedPortTCP1.Bindings.Add.IP:= '127.0.0.1'; IdMappedPortTCP1.Bindings.Add.Port := 8080; ** IdMappedPortTCP1.MappedHost := remote_proxy; IdMappedPortTCP1.MappedPort := remote_port;** IdMappedPortTCP1.Active := True` How do you change the mapped host to make it connect to the server the client asks for instead of the mapped host? An example code would be very helpful – Wanyiri Sep 19 '15 at 14:57
  • first, you are calling `Bindings.Add` too many times. Call it *once* and then set the IP *and* port of the new binding. Second, in the `OnConnect` event, read the client's request from `AThread.Connection`, parse the requested host from it, and set the `AThread.OutboundClient.Host` accordingly. – Remy Lebeau Sep 19 '15 at 16:21
  • Please explain what you mean by call the `Bindings.Add` once and then set IP and Port, am still learning these things. In the `OnConnect` event, I was able to parse the requested host by assigning the `AThread.Connection.CurrentReadBuffer` to a `TIdHeaderList` and `myhost := myheaders.Values['host'];`. But I have no idea how to get the port from the request. also is this the best way to get the host and port? I also want to enquire, since I will be using dynamic hosts and ports, do i still need to set the `MappedHost` and `MappedPort` to my `IdMappedPortTCP1` or can i just omit them? – Wanyiri Sep 20 '15 at 01:15
  • `Bindings.Add` creates a new listening socket on the server. Each binding represents a separate listening socket. When you call `Add` twice, you are creating 2 listening sockets - one that you are assigning an `IP` to but no `Port` (so it will use the `DefaultPort`), and the other that you are assigning a `Port` to but no `IP` (so it will listen on all local IPs). `IdMappedPortTCP1.Bindings.Add.IP:= '127.0.0.1'; IdMappedPortTCP1.Bindings.Add.Port := 8080;` should be `with IdMappedPortTCP1.Bindings.Add do begin IP:= '127.0.0.1'; Port := 8080; end;` instead. – Remy Lebeau Sep 20 '15 at 02:11
  • You should NOT be using `AThread.Connection.CurrentReadBuffer` to populate the `TIdHeaderList`. `CurrentReadBuffer` is *RAW DATA*, it has no structure to it. Use `AThread.Connection.Capture()` instead, specifying `''` as the `ADelim` parameter, so that you read the headers line-by-line until the header delimiter is reached, then process the headers and finish reading the rest of the HTTP request body as described by the headers... – Remy Lebeau Sep 20 '15 at 02:14
  • The `Host` header is NOT guaranteed to be present unless the client is sending an HTTP 1.1 request. So, if the client does not send a `Host` header, and does not send a `CONNECT` request, and does not specify a full URL in the request, you will not know which host the client is trying to connect to. The last two conditions usually only occur if the client explicitly knows that it is connecting to an HTTP proxy server, so it can provide information about the intended destination server. If the client *DOES* send a host, the port defaults to 80 (HTTP) or 443 (HTTPS) unless stated otherwise. – Remy Lebeau Sep 20 '15 at 02:16
  • And yes, you can omit the `MappedHost` and `MappedPort` if you are providing the host/port dynamically. But you have to be sure to provide it every time, or else you cannot make a connection. – Remy Lebeau Sep 20 '15 at 02:19
  • Update: I used `AThread.Connection.Capture(myheaders,'');` so now i can assign my host to `myheaders.Values['host']` and if `AThread.Connection.ReadLn = 'CONNECT'` I set port to 443 otherwise I set it as 80. Am I on the right track? – Wanyiri Sep 20 '15 at 11:13
  • `var myheaders:TIdHeaderList; method : string; begin myheaders:=TIdHeaderList.Create; try Athread.Connection.Capture(myheaders); method:= AThread.Connection.ReadLn; if myheaders.Count<>0 then begin if method='CONNECT' then begin with TIdTCPClient(AThread.OutboundClient) do begin Host:=myheaders.Values['host']; Port:=443; end; end else begin with TIdTCPClient(AThread.OutboundClient) do begin Host:=myheaders.Values['host']; Port:=80; end; end;` – Wanyiri Sep 20 '15 at 12:24
  • cont'd `TIdMappedPortThread(AThread).NetData:= method + #13#10 + myheaders.Text + #13#10 + #13#10; end else begin TIdMappedPortThread(AThread).NetData:= method + #13#10 + #13#10; outs.Lines.Add(TIdMappedPortThread(AThread).NetData); end; finally myheaders.Free; end; end;` I have put that code in the `OnConnect` event but it does not seem to be working..What have i done wrong? – Wanyiri Sep 20 '15 at 12:25
  • First, don't put large code snippets in comments. Edit your question instead (which I have now done for you). Second, your code is backwards. Call `ReadLn()` first to read the request line (`CONNECT`, `GET`, etc), then call `Capture()` to read the request headers. And you are making incorrect assumptions about the port and the formatting of the request line. Did you read the HTTP specifications yet? – Remy Lebeau Sep 20 '15 at 16:39
  • And lastly, this all hinges on the fact that the client needs to be configured to connect to your server as an HTTP proxy server so it can adjust its HTTP requests accordingly, otherwise this whole effort is useless. Indy servers can only receive connections that are explicitly made to them by clients. If the client cannot be configured as such, you have to use something like Proxifier to redirect it to your proxy. – Remy Lebeau Sep 20 '15 at 16:43
  • I made a few corrections in the code, calling the `ReadLn()` before `Capture()` and then checking for connection method in the request line. I have set my client to connect via the proxy . I also hardcoded the ports for testing purposes. What is wrong with the code since its not working? Also, the client sends a Host header similar to `www.example.com:80` a combination of Host:Port, but when calling retrieving the `Host:=myheaders.Values['host']` it only gets only the host address part, is there a way to get the port as well such that i can assign the `Port` dynamically as well? – Wanyiri Sep 21 '15 at 12:24
  • First, use `Capture(myheaders, '')` and not `Capture(myheaders)`. The default terminator is `'.'`, which is not correct for HTTP. Second, `Values[]` returns the complete header, there is no way it can strip off the port if present, and there is no reason for the `Host` to ever include `:80` since that is the default port for HTTP. – Remy Lebeau Sep 21 '15 at 17:51
  • Third, you have clearly not read the relevant HTTP specs. DO NOT use the `Host` header *at all* when implementing an HTTP proxy. The request line tells you where to connect. For `CONNECT`, the request line will look like `CONNECT host:port HTTP/1.0`. For other requests (`GET`, etc), the request line will look more like `GET http://host:port/path HTTP/1.0` instead. You need to parse the request line accordingly to get the host and port. In the case of `CONNECT`, you would make the connection, send back a `200` reply, and then let `TIdMappedPortTCP` handle the rest of the data for you. – Remy Lebeau Sep 21 '15 at 17:53
  • But for any other request, you would have to make the connection, retrieve the requested resource from the target server, and send an appropriate reply with the data back to the client. – Remy Lebeau Sep 21 '15 at 17:54
  • As I told you earlier, `TIdMappedPortTCP` is NOT really well-suited for implementing an HTTP proxy, as you have to manually implement a full HTTP protocol handler on top of it. Indy 10 has a `TIdHTTPProxyServer` component that manages those details for you, but Indy 9 does not have `TIdHTTPProxyServer`. If you cannot upgrade to Indy10, I would suggest switching to `TIdHTTPServer` and `TIdHTTP` instead of using `TIdMappedPortTCP`. – Remy Lebeau Sep 21 '15 at 17:56
  • BTW, here is the `CONNECT` spec: [Tunneling TCP based protocols through Web proxy servers](https://tools.ietf.org/html/draft-luotonen-web-proxy-tunneling-01), which is separate from the main HTTP spec: [RFC 2616](http://tools.ietf.org/html/rfc2616) (which was later obsoleted by RFCs 7230 - 7235). – Remy Lebeau Sep 21 '15 at 17:57
  • Indy 10's `TIdHTTPProxyServer` requires installing of root cert to modify https headers which is which I am trying to avoid. If i switch to `TIdHTTPServer` and `TIdHTTP ` wouldn't i still need to install a root cert to modify headers for https request? `TIdMappedPortTCP` though not well suited, works well without the need for a root cert, but the only way i can use it for now is connecting via a proxy, isn't there a simpler way of avoiding a proxy? instead of client > `TIdMappedPortTCP` > proxy > remotehost, to connect something like client > `TIdMappedPortTCP` > remotehost ? – Wanyiri Sep 21 '15 at 19:18
  • You CANNOT modify HTTPS data if the client does not create a secure session directly with your server. If your server does not have that session (if the client merely passes through your server, such as in the case of `CONNECT`, to create a secure session with the remote server instead, as it should be), your server CANNOT decrypt/modify/encrypt the HTTPS data. If it were possible to do so, SSL/TLS security would be compromised. `TIdHTTPProxyServer` and `TIdMappedPortTCP` work EXACTLY the same way in this regard. I do not think you understand how HTTPS and proxies actually work. – Remy Lebeau Sep 21 '15 at 19:34
  • A `CONNECT` request to an HTTP proxy does not require a proxy certificate, but the proxy cannot modify HTTPS data, as it is just acting as a tunnel of raw bytes and thus will only see encrypted data without any encryption keys to access it. A proxied HTTPS request like `GET`, on the other hand, needs a proxy certificate, but the proxy can modify HTTPS data, because the client is creating a secure session with the proxy directly, and then the proxy is creating a secure connection to the remote server, accepting requests from the client and sending its own requests to the remote server. – Remy Lebeau Sep 21 '15 at 19:36
  • Interesting. Thanx for the explanation. As I am not intending to decrypt HTTPS data, does that mean I can implement a proxy using `TIdHTTPProxyServer`, modify the request headers and make a secure connection to the intended remote server without having to install a certificate? – Wanyiri Sep 21 '15 at 19:52
  • HTTPS is encrypted, headers and all. The only way you can modify the headers is if you decrypt the data, which means the client needs a secure session with your proxy, which needs a certificate (it can be a self-signed certificate, as long as the client is willing to accept it). The client sends encrypted data to your proxy, which it decrypts and modifies, then creates a separate secure session with the remote server and sends the decrypted data using the remote server's encryption, and vice versa for responses. That only works for non-`CONNECT` requests like `GET`. You are SOL with `CONNECT`. – Remy Lebeau Sep 22 '15 at 17:07
  • Just a quick question, can you. is the `TIdMappedPortTCP`'s `OnExecute` event triggered before or after the `OnConnect`. Is it possible to change the `AThread.OutboundClient.Host` and `AThread.OutboundClient.Host` in it? – Wanyiri Sep 22 '15 at 22:42
  • You could have tested that yourself in a matter of seconds instead of asking here. The event sequence is: `OnBeforeConnect` -> `OnConnect` -> `OnOutboundConnect` -> `OnExecute`/`OnOutboundData` -> `OnDisconnect/`OnOutboundDisconnect`. And yes, you can customize the `OutboundClient` in the `OnConnect` event (otherwise this whole exercise would have been useless). Now please, this discussion has dragged on for FAR too long. StackOverflow is not the place for lengthy comment chains. Either post new questions, or move over to some other discussion forums. – Remy Lebeau Sep 23 '15 at 05:01
  • I was asking if you could Change the `AThread.OutboundClient.Host` in the `OnExecute` NOT in the `OnConnect ` but u answered anyway. Though my problem remains unsolved, I now have more information to work on towards that. Thanks @RemyLebeau for all the insights and alternatives you have given. – Wanyiri Sep 23 '15 at 10:54
  • `OnBeforeConnect` is too early (`OutboundClient` has not been created yet), and `OnExecute` is too late (data is already flowing). Customizing the `OutboundClient` can only be done in the `OnConnect` event, which is triggered just before `OutboundClient.Connect()` is called. – Remy Lebeau Sep 23 '15 at 16:59

0 Answers0