7

I am writing domain checker for a website but i face funny issue.

Some whois server which responsible for a domain like .WS (whois.worldsite.ws) or .CA (whois.cira.ca) Block ip address after 2 or 3 time query per day !!!

It is look like disaster for a hosting website which may have million request per day.

What should i do to solve this problem ?

  • 3
    Whois servers are rate limited for a good reason, but what you report is kind of very very low, are you sure that you are blocked with as little as 2 or 3 queries? I doubt it. There is probably other traffic coming from the same IP that may trigger defensive measures at the other endpoint. – Patrick Mevzek Jan 03 '18 at 21:38

4 Answers4

14

On the contrary there's a very easy way around it.

As has been said most whois authorities will throttle (or even block) your traffic if they deem that your making too many requests in a 24 hour period, instead you might want to consider logging in to the ftp site of any of the whois providers worldwide and downloading the various bits of the database, then writing (or finding) your own script to process them.

I currently do that with one of my own servers, which connects using the following shell script (once every 24 hours):

#!/bin/bash
rm -f delegated-afrinic-latest
rm -f delegated-lacnic-latest
rm -f delegated-arin-latest
rm -f delegated-apnic-latest
rm -f delegated-ripencc-latest
rm -f ripe.db.inetnum
rm -f apnic.db.inetnum
rm -f ripe.db.inetnum.gz
rm -f apnic.db.inetnum.gz
wget ftp://ftp.afrinic.net/pub/stats/afrinic/delegated-afrinic-latest
wget ftp://ftp.lacnic.net/pub/stats/lacnic/delegated-lacnic-latest
wget ftp://ftp.arin.net/pub/stats/arin/delegated-arin-latest
wget ftp://ftp.apnic.net/pub/stats/apnic/delegated-apnic-latest
wget ftp://ftp.ripe.net/ripe/stats/delegated-ripencc-latest
wget ftp://ftp.ripe.net/ripe/dbase/split/ripe.db.inetnum.gz
ftp -n -v ftp.apnic.net <<END
user anonymous anonymous@anonymous.org
binary
passive
get /apnic/whois-data/APNIC/split/apnic.db.inetnum.gz apnic.db.inetnum.gz
bye
END
gunzip ripe.db.inetnum
gunzip apnic.db.inetnum

I then have a custom written program that parses the files out into a custom database structure which my servers then do their queries from.

Since all the servers mirror each others data, then you should be able to get a full data set from one server, but if not, then it wouldn't take much to modify the above shell script to download the data from the other servers, all of them respond to 'ftp.????' and have the same universal folder structure.

I can't help you with the parser however as that contains proprietary code, but the file format (esp if you get the split files) is identical to what you see in a typical whois output so it's very easy to work with.

There is a parser on google-code (That's where I got the download script) called 'ip-country' (I think) it's designed to allow you to build your own whois database, the one I've built is slightly more complicated as it's combined with other data too (Hence why my parser is proprietary)

By downloading and processing your own data like that, you get around any limit imposed by the providers, and the upshot is that it's most likely way faster to query your own data store than keep firing off requests from your server to the query servers every time someone enters an IP address.

shawty
  • 5,729
  • 2
  • 37
  • 71
  • Thank you for the idea and sample script. – Xeoncross Nov 30 '14 at 23:02
  • Your most welcome, I hope it proves to be of some use. Parsing the data is no fun however, believe me, the file format is horrible :-) – shawty Dec 01 '14 at 11:18
  • That would be nice :-) , you could try taking a look at : http://www.realip.info/api/ : see if anything there is useful. – shawty Dec 01 '14 at 17:00
  • Very good clue but please describe more about files and what exactly they contains they seam have really irreverent data. I cant how can extract whois data from. – MSS Mar 13 '17 at 16:44
  • 1
    They are all different, the different authorities around the world all use different formats for the files, so you'll basically have to do an analysis on what's what in each file and figure out how to carve it up. There is no "standard way" that this works, that's why there are no libraries or pre-made code available. If there was, chances are it would only work one time then fail the rest. Only way to do this, is with long hours reading the file again and again and picking it apart one character at a time to get what you need. – shawty Mar 14 '17 at 09:22
  • 2
    What you show is RIRs data, not domain name data, so completely irrelevant to the question. – Patrick Mevzek Jan 03 '18 at 21:36
  • And what I typed, way back in 2013, when I originally answered this question was actually valuable to someone about a year later, and NO ONE except you has complained about anything since... so with the greatest respect, please just go away if you have nothing that's of any constructive use to say. – shawty Jan 06 '18 at 17:57
  • Hey @shawty, could you share the program you wrote for parsing the data? I would definitely find that useful - like your answer, even in 2018. ;) – mdt Jan 31 '18 at 13:45
  • Sorry, I'm afraid I can't. I originally wrote it for a client of my business, and once finished the code was handed over to them. I no longer have the code, and I have no permission to re-use the code either. – shawty Jan 31 '18 at 17:47
  • These files clearly are not maintained. Example of obvious gaps after scanning the files: 3.0.0.0-4.255.255.255 5.45.36.0/22 6.0.0.0-8.8.4.3 8.8.4.5-8.8.8.7 8.8.8.9-8.127.255.255 8.192.0.0/12 8.224.0.0-9.255.255.255 Doing a regular whois query on these ips they are looked up correctly... – jjxtra Apr 16 '22 at 21:50
  • 1
    Well that's an issue for you to take up wish the various registry authorities then ;-) – shawty Apr 19 '22 at 22:03
1

There is no easy way round this. Some whois servers as you have found out block you after a couple of queries. You can usually pay them a fee for a set amount of queries. There is another way that may work. Since the blocking is done by IP address you could write a Java applet which would run the query. That way the limit would be on the user so the user would be limited to 2-3 queries per day rather than the web site. The Java Applet would have to be signed to allow network access. I wrote a whois library in java that may be of help JFreeWhois

ozkanpakdil
  • 3,199
  • 31
  • 48
Art Vandelay
  • 111
  • 3
0

You should know error message patterns and try to repeat queries after some sleep. If it does not work then use different Whois server. As you are writing application for a website, Whois query should be done from client side.

Justinas Jakavonis
  • 8,220
  • 10
  • 69
  • 114
0

You can use proxies like TOR and here is a good approach another way would be using torsocks with whois client. Or you can use I2P https://unix.stackexchange.com/a/340568/444120 too.

Also whois services has paid options, you can contact with their support.

ozkanpakdil
  • 3,199
  • 31
  • 48