2

I try to set up an HTTP connection using python, using the following code. I do the DNS resolving myself because I cannot change things like the hosts and resolv.conf files on the machines in question.

class resolver(object):
    def __init__(self, server):
        self.server = server
    def __call__(self, host):
        res = check_output('host %s %s' % (host, self.server), shell = True)
        ips = findall('has address (\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})', res)
        return choice(ips)

class http_connection(httplib.HTTPConnection):
    dns_server = '8.8.8.8'
    def connect(self):
        resolv = resolver(http_connection.dns_server)
        self.sock = socket.create_connection((resolv(self.host), self.port), self.timeout)
...
x = http_connection('%s:%d' % (args.host, args.port))
x.request('GET','/start/')
rep = x.getresponse()
print rep.read()

Everything works, I get an IP from DNS, the socket gets created (with an ip-port pair, not a hostname) and I do get an answer. The only thing is, it takes a long time, up to 5-6 seconds. After debugging I found out that for some reason a reverse DNS lookup for the resolved IP happens, which times out. I verified this was the problem by reproducing it on another machine, after adding an entry to /etc/hosts everything worked as fast as I'd expect. Also, for what it's worth: using curl or wget also resulted in the RDNS.

I see two options to get around this but can't find out how to actually implement them:

  1. Ideally, I do want a reverse-lookup to match the default behaviour as much as possible, so is there a way to force this lookup to go to the dns_server I provide myself (which is not in resolv.conf).
  2. If that is not possible, can I disable this RDNS somewhere? I can go up to the setsockopt level.
KillianDS
  • 16,936
  • 4
  • 61
  • 70
  • 1
    Have you tried with just `socket.connect()` instead? – Nikolai Fetissov Nov 07 '12 at 18:05
  • Yes, good notion, but I did try to set up the socket connection myself instead of using `create_connection`, because I noticed a `getaddrinfo` in there. I got the same result. I found a way to circumvent it however, will post the answer. – KillianDS Nov 08 '12 at 09:42

1 Answers1

1

I finally found a way to get around this myself. Apparently this happens when I do not explicitly provide a source address myself and let the system choose. I imagine Linux does some tricks to find out which interface is best to use and RDNS is one of those.

By providing a source_address as follows, everything works fine:

class http_connection(httplib.HTTPConnection):
    def __init__(self, dns_server, *args, **kwargs):
        httplib.HTTPConnection.__init__(self, *args, **kwargs)
        self.resolv = resolver(dns_server, self.source_address)
    def connect(self):
        self.sock = socket.create_connection((self.resolv(self.host), self.port), self.timeout, self.source_address)
...
x = http_connection(args.dns_server, '%s:%d' % (args.host, args.port), source_address = (args.source_address, 0))
KillianDS
  • 16,936
  • 4
  • 61
  • 70