EventMachine and Twitter streaming API

Question

I am running an EventMachine process using the Twitter streaming API. I always have an issue if the content of the stream is not frequently.

Here is the minimal version of the script:

require 'rubygems'
require 'eventmachine'
require 'em-http'
require 'json'

usage = "#{$0} <user> <password> <track>"
abort usage unless user = ARGV.shift
abort usage unless password = ARGV.shift
abort usage unless keywords= ARGV.shift

def startIt(user,password,keywords)
EventMachine.run do
  http = EventMachine::HttpRequest.new("https://stream.twitter.com/1/statuses/filter.json",{:port=>443}).post(
                    :head =>{ 'Authorization' => [ user, password ] } , 
                    :body =>{"track"=>keywords},
                    :keepalive=>true,
                    :timeout=>-1)

  buffer = ""
  http.stream do |chunk|
    buffer += chunk
    while line = buffer.slice!(/.+\r?\n/)
      if line.length>5
          tweet=JSON.parse(line)
          puts Time.new.to_s+"#{tweet['user']['screen_name']}: #{tweet['text']}"
      end
    end

  end
   http.errback {
        puts Time.new.to_s+"Error: "
        puts http.error
   }
end  
    rescue => error
      puts "error rescue "+error.to_s
end

while true
    startIt user,password,keywords
end

If I search for a keyword like "iphone", everything works well If I search for a less frequently used keyword, my stream keeps to be closed very rapidely , around 20 sec after the last message. Note: that http.error is always empty, so it's very hard to understand while the stream is closed... On the other end, the nerly similar php version is not closed, so seems probably in issue with eventmachine/http-em but I dont' understand which one...

Any idea of how you could dynamically add/remove keywords in this example? — tibbon, May 18 '12 at 16:57
@tibbon as far as I know, the only way to add/remove keyword is to stop the stream. To do it without loosing any tweet, the best way is to handle several stream. For instance, opening a second stream with the new parameters (and a different twitter account) and once opened, closing the first one. But be careful of possible duplication during the small period where both stream are opened, especially in case of high volume — tomsoft, May 19 '12 at 08:37
hey @tomsoft I am using similar code like you but for me `http.stream do |chunk| puts chunk end` doesnt print anything ...it just hangs there ..any idea? — Rahul Dess, May 19 '16 at 06:54

score 6 · Accepted Answer · answered Jan 22 '12 at 19:35

6

You should add settings to prevent your connection to timeout. Try this :

http = EventMachine::HttpRequest.new(
  "https://stream.twitter.com/1/statuses/filter.json",
  :connection_timeout => 0,
  :inactivity_timeout => 0
).post(
  :head => {'Authorization' => [ user, password ] } , 
  :body => {'track' => keywords}
)

Good luck, Christian

answered Jan 22 '12 at 19:35

Chris

2,744
3
24
39

Hey @chris I am using similar code like you but for me `http.stream do |chunk| puts chunk end` doesnt print anything ...it just hangs there ..any idea? – Rahul Dess May 19 '16 at 06:54

EventMachine and Twitter streaming API

1 Answers1