Ruby async whois

Question

I'm using whois to perform and parse whois queries. The problem is: i need to perform much more. Thousands and thousands. I found async package em-whois, which sounds perfect for me. This is simple addon for whois, which relies on Ruby Fibers and em-synchrony.

My current code to perform whois queries looks like this:

...
c = Whois::Client.new(:timeout => timeout)
r = c.query(domain)
...

After I've installed em-whois, i'm trying to extend this basic example.

My code looks like this now, and still is very slow:

$: << File.dirname(__FILE__) + '/../lib'
require 'em-whois'
EM.synchrony do  
  domain = ARGV[0] || "github.com"
  $i = 0;
  $num = 100;
  arr = Array.new($num)
  # perform all queries
  begin
     puts("Inside the loop i = #$i" );
     $i += 1;
     arr[$i] = Whois.whois(domain);
  end while $i < $num
  $i = 0;
  # get all results
  begin
     $i += 1;
    puts "\r#{domain}\n#{"-" * domain.length}"
    [:available?, :status, :expires_on].each do |k|
      puts "#{k}: #{arr[$i].properties[k]}"
    end
  end while $i < $num
  EM.stop
end

How do i perform batch queries (1k of domains) using async power of whois and em-whois ?

I don't have any experience with ruby before. I'm Python/C/PHP developer. Please help.

Opening up multiple simultaneous connections to a whois server seems like a good way to get permanently blocked. — pguardiario, Sep 27 '12 at 07:02
my question is more theoretical. i need help with code, i'm aware of limitation — artyomboyko, Sep 27 '12 at 07:13
But it's quite possible that the server is throttling you, and your code is fine. If you profile the code, does it sit a lot in I/O wait? — tripleee, Sep 27 '12 at 08:01
You need help with more than the code. The concept is flawed. — pguardiario, Sep 27 '12 at 09:26

score 2 · Accepted Answer · answered Sep 27 '12 at 20:51

Answer from author of the library. Example linked here.

$: << File.dirname(__FILE__) + '/../lib'
require 'em-whois'
require 'atomic'

# Asynchronous, parallel multi-domain WHOIS lookup
domains = ARGV.empty? ? ["github.com", "google.com", "bing.com", "yahoo.com", "mikejarema.com"] : ARGV
whois   = {}

EM.synchrony do  

  # Progress indicator
  EM.add_periodic_timer(0.1) do
    STDERR.print "."
  end

  # Exit condition
  EM.add_periodic_timer(0.1) do
    if domains.size == whois.keys.size
      puts ""
      whois.each do |k,v|
        if v.properties[:available?]
          puts "#{k}: available"
        else
          puts "#{k}: taken, expires #{v.properties[:expires_on]}"
        end
      end

      EM.stop
    end
  end

  # Async WHOIS lookup - fire and forget via parallel fibers
  domains.each do |i|
    Fiber.new do
      whois[i] = Whois.whois(i)
    end.resume
  end

end

I'm the author of em-whois and provided the above code, and want to be clear I strongly frown on unauthorized WHOIS-data harvesting. That said, there are two caveats to the above solution: 1) it fires ALL whois queries at approx the same time, further work can be done to control the request frequency (eg. 100/s) 2) this script quickly hits WHOIS rate limits when run from a single IP, proxying across IPs is another consideration. — Mike Jarema, Sep 28 '12 at 15:51
Third caveat: 3) in production use-cases, exceptions should be handled within the "fire and forget" fibers, otherwise the exit condition may never be satisfied. — Mike Jarema, Sep 28 '12 at 15:57

Ruby async whois

1 Answers1