9

If say I want to retrieve a web page for parsing, but not block the CPU while the I/O is taking place. Is there something equivalent to Python's Eventlet library?

Fast Fish
  • 385
  • 1
  • 3
  • 7

2 Answers2

18

The best HTTP client library for Ruby is Typhoeus, it can be used to perform multiple HTTP requests in parallel in a non-blocking fashion. There is a blocking and non-blocking interface:

# blocking
response = Typhoeus::Request.get("http://stackoverflow.com/")
puts response.body

# non-blocking
request1 = Typhoeus::Request.new("http://stackoverflow.com/")
request1.on_complete do |response|
  puts response.body
end
request2 = Typhoeus::Request.new("http://stackoverflow.com/questions")
request2.on_complete do |response|
  puts response.body
end
hydra = Typhoeus::Hydra.new
hydra.queue(request1)
hydra.queue(request2)
hydra.run # this call is blocking, though

Another option is em-http-request, which runs on top of EventMachine. It has a completely non-blocking interface:

EventMachine.run do
  request = EventMachine::HttpRequest.new('http://stackoverflow.com/').get
  request.callback do
    puts request.response
    EventMachine.stop
  end
end

There's also an interface for making many requests in parallel, similarly to Typhoeus Hydra.

The downside of em-http-request is that it is tied to EventMachine. EventMachine is an awesome framework in itself, but it's an all-or-nothing deal. You need to write your whole application in an evented/continuation-passing-style fashion, and that has been known to cause brain damage. Typhoeus is much better suited to applications that are not already evented.

alex
  • 158
  • 6
Theo
  • 131,503
  • 21
  • 160
  • 205
  • 1
    When you say the call to hydra.run is blocking, that is fine, so long as it remains in a sleep state, and is woken when the I/O is complete. This is what I'm looking to achieve, just like with event driven I/O in Windows. Whatever thread the hydra.run call is made on should be taking no CPU while it blocks, because essentially it is waiting on an event. Is that how hydra works? If not, I'd think it sort of defeats the purpose. If you could confirm, I'll mark this as accepted. Thanks. – Fast Fish Dec 20 '10 at 04:28
  • In my case, I need to release the current thread and not block at all. Is there any way to avoid the blocking 'hydra.run' call? Are there any other HTTP ruby gems that support a completely non-blocking approach? – Jeremy Haile Nov 21 '14 at 19:15
  • I think you can do it with Celluloid::IO, for example: https://github.com/httprb/http.rb/wiki/Parallel-requests-with-Celluloid%3A%3AIO – Theo Jan 10 '15 at 10:02
  • Celluliod is nicely wrapped by the Suckerpunch gem, which lets you kick off worker tasks (which can make external calls if you like) that run in the background. (Actually, it looks like Suckerpunch switched over to using Concurrent Ruby instead of Celluloid late last year.) If the process is terminated, you'll lose the workers. You may need to allow for that. – A Fader Darkly Feb 01 '16 at 14:43
  • Or you could use Concurrent Ruby directly. – A Fader Darkly Feb 01 '16 at 14:43
5

I'm not sure what Eventlet does, but Ruby has EventMachine, a library for non-blocking IO (amongst other things).

maerics
  • 151,642
  • 46
  • 269
  • 291