Can this be optimized? whois over a list

Question

This code in Python 3.4 works fine, but I am wondering how it can be made faster:

from string import ascii_lowercase
from itertools import product
import whois

domains = list()
available_domains = list()

for c in map(''.join, product(ascii_lowercase, repeat=2)):
    domains.append(c+'.at')

for dom in domains:
    domain = whois.query(dom)
    if domain == '':
        available_domains.append(domain)

print(available_domains)

Any input will be appreciated. Thanks!

This is probably better suited for [CodeReview](https://codereview.stackexchange.com) — UnholySheep, Oct 26 '17 at 09:18
I would guess that `whois.query(dom)` is where your program spends most of the time (network call), and if that's true - what can you do to improve it? — Nir Alfasi, Oct 26 '17 at 09:19
whois is fetching data from the internet, commonly domain servers will deliberately throttle non-vital queries as a protection against dds type attacks. If you want to run it faster, you could bring all the websites you're whois-ing in-house onto your own private lan and optimise them for this query. (minor sarcasm) — Thomas Kimber, Oct 26 '17 at 09:20
My suggestion is to use the `async` module to execute the `whois.query` call asynchronously. — Ignacio Vergara Kausel, Oct 26 '17 at 09:20
Code aside, lack of a WHOIS record does not necessarily imply that a domain is available. Also, you're going to get blocked by most WHOIS servers very quickly by running a lot of queries. — Alex Riley, Oct 26 '17 at 09:23

score 1 · Answer 1 · answered Oct 26 '17 at 09:26

1

Well you can get domains with comprehension:

domains = [''.join(i) + '.at' for i in product(ascii_lowercase, repeat=2)]

And do the same with available_domains:

available_domains = [d for d in domains if whois.query(d)]

At least it is fewer lines :)

answered Oct 26 '17 at 09:26

zipa

27,316
6
40
58

score 0 · Answer 2 · answered Oct 26 '17 at 09:21

You could use the multiprocessing module to map over the domains list, to be able to do the queries in parallel instead of serially. (By the way, since whois.query() is likely IO-bound, you can use the multiprocessing.dummy module to use threads instead of subprocesses.)

Can this be optimized? whois over a list

2 Answers2