0

Following code snippet fetches multiple public DNS servers asynchronously. If script is executed in PyCharm it works perfectly and resolves all given resolvers with very few errors (~14 errors in 1078 requests).

However if I run exactly same script in OS X terminal only first ~280 aiodns requests are successful and the rest returns aiodns.DNSError(11, 'Could not contact DNS servers')(~834 errors in 1078 requests).

Copy/paste resolvers_short list from https://pastebin.com/wSYtzebZ

This code is part of my open-source project on https://github.com/MMquant/DNSweeper/blob/master/DNSweeper.py

import asyncio
import aiodns

#resolvers_short = [fill resolvers from link]

class Fetcher(object):

    def __init__(self):
        self.loop = asyncio.get_event_loop()

    def get_records(self, names, query_type, resolvers):

        coros = [self._query_sweep_resolvers(names, query_type, resolver) for resolver in resolvers]
        tasks = asyncio.gather(*coros, return_exceptions=True)

        records = self.loop.run_until_complete(tasks)

        return records

    async def _query_sweep_resolvers(self, name, query_type, nameserver):

        resolver = aiodns.DNSResolver(
            nameservers=[nameserver],
            timeout=5,
            tries=3,
            loop=self.loop
        )

        try:
            result = await resolver.query(name, query_type)
        except aiodns.error.DNSError as e:
            result = e

        return {'ns': nameserver,'name': name ,'type': query_type, 'result': result}


def errors_count(results):

    count = 0
    for result in results:
        if type(result['result']) is aiodns.error.DNSError:
            count += 1
    return count


if __name__ == '__main__':

    fetcher = Fetcher()
    results = fetcher.get_records('www.flickr.com', 'A', resolvers_short)
    # print(results)
    errors = errors_count(results)
    # In 1078 resolvers
    # If script executed in PyCharm there are only ~14 aiodns response errors on average
    # If script executed in terminal there are ~834 aiodns response errors where majority are
    # DNSError(11, 'Could not contact DNS servers')
    print(errors)
    pass

I have no idea how to continue with debugging.

These are modules which I'm using:

aiodns==1.1.1
pycares==2.3.0
Petr Javorik
  • 1,695
  • 19
  • 25

1 Answers1

0

I found out that OS X allows just 256 TCP connections at once. I increased it by editing open file descriptors soft limit 

$ ulimit -S -n
256
$ ulimit -S -n 50000
$ ulimit -S -n
50000

I additionally added following code to dynamically set open file descriptors soft limit from the program:

import resource

new_soft_limit = 50000

(rlimit_nofile_soft, rlimit_nofile_hard) = resource.getrlimit(resource.RLIMIT_NOFILE)
resource.setrlimit(resource.RLIMIT_NOFILE, (new_soft_limit, rlimit_nofile_hard))
Petr Javorik
  • 1,695
  • 19
  • 25