0

I'm trying to determine high anonymity proxies. Also called private/elite proxies. From a forum I've read this:

High anonymity Servers don't send HTTP_X_FORWARDED_FOR, HTTP_VIA and HTTP_PROXY_CONNECTION variables. Host doesn't even know you are using proxy server and of course it doesn't know your IP address.

A highly anonymous proxy will display the following information:

REMOTE_ADDR = Proxy's IP address

HTTP_VIA = blank

HTTP_X_FORWARDED_FOR = blank

So, how I can check for this headers in Python, to discard them as a HA Proxy ? I have tried to retrieve the headers for 20-30 proxies using the requests package, also with urllib, with the build-in http.client, with urllib2. But I didn't see these headers, never. So I should be doing something wrong...

This is the code I've used to test with requests:

proxies = {'http': 'http://176.100.108.214:3128'}
header = {'user-agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.360',}
s = requests.session()
s.proxies = proxies
r = s.get('http://www.python.org', headers=header)
print(r.status_code)
print(r.request.headers)
print(r.headers)
Jeflopo
  • 2,192
  • 4
  • 34
  • 46

1 Answers1

0

It sounds like the forum post you're referring to is talking about the headers seen by the server on your proxied request, not the headers seen by the client on the proxied response.

Since you're testing with www.python.org as the server, the only way to see the headers it receives would be to have access to their logs. Which you don't.

But there's a simple solution: run your own HTTP server, make requests against that, and then you can see what it receives. (If you're behind a firewall or NAT that the proxy you're testing won't be able to connect to, you may have to get a free hosted server somewhere; if not, you can just run it on your machine.)

If you have no idea how to set up and configure a web server, Python comes with one of its own. Just run this script with Python 3.2+ (on your own machine, or an Amazon EC2 free instance, or whatever):

from http.server import HTTPServer, SimpleHTTPRequestHandler

class HeaderDumper(SimpleHTTPRequestHandler):
    def do_GET(self):
        try:
            return super().do_GET()
        finally:
            print(self.headers)

server = HTTPServer(("", 8123), HeaderDumper)
server.serve_forever()

Then run that script with python3 in the shell.

Then just run your client script, with http://my.host.ip instead of http://www.python.org, and look at what the script dumps to the server's shell.

abarnert
  • 354,177
  • 51
  • 601
  • 671
  • Ok I've understood the client/server trick. And I've remembered the $_SERVER[] PHP var (lol) But That server you wrote would be listening all the time for a new get request. On 127.0.0.1:8123 I guess. It's what the HeaderDumper() does, right ? – Jeflopo May 18 '15 at 00:48
  • @JesúsFlores: Well, it listens on `0.0.0.0:8123`, not just `127.0.0.1:8123`, which is why the proxy can connect to it (assuming it's on a reachable IP address). But otherwise, yes, it's a server that listens forever for new get requests (and responds with a directory listing of its current directory), and, in addition to printing the normal `127.0.0.1 - - [17/May/2015 17:49:49] "GET / HTTP/1.1" 200 -` log line to the terminal, it also prints out all the headers it sees. – abarnert May 18 '15 at 00:50