Storing tweets using tweetstream in MongoDB fails with following error

Question

I have just started to learn ruby and concept of mongodb. This is the script that I am trying to run

require 'rubygems'
require 'tweetstream'
require 'mongo'

TweetStream.configure do |config|
  config.consumer_key = '<key>'
  config.consumer_secret = '<secret>'
  config.oauth_token = '<token>'
  config.oauth_token_secret = '<token_secret'
  config.auth_method = :oauth
end

connection = Mongo::Connection.new

db = connection.db("tweetsDB")

tweets = db.collection("tweets")

client = TweetStream::Client.new

client.on_error do |message|
  puts message
end

client.follow(<user_id>,<user_id>) do |status|
        id = tweets.insert(status, :safe => true)
end

NOTE: I have removed all the static private values with in the script above for this post.

Version of Mongo, bson, bson_ext - 1.7.0

error message

NoMethodError: undefined method `has_key?' for #<Twitter::Tweet:0x7f21cd14cf08>
    /var/lib/gems/1.8/gems/bson-1.7.0/lib/bson/types/object_id.rb:93:in `create_pk'
    /var/lib/gems/1.8/gems/mongo-1.7.0/lib/mongo/collection.rb:360:in `insert'
    /var/lib/gems/1.8/gems/mongo-1.7.0/lib/mongo/collection.rb:360:in `collect!'
    /var/lib/gems/1.8/gems/mongo-1.7.0/lib/mongo/collection.rb:360:in `insert'
    tracker.rb:28
    /var/lib/gems/1.8/gems/tweetstream-2.3.0/lib/tweetstream/client.rb:525:in `call'
    /var/lib/gems/1.8/gems/tweetstream-2.3.0/lib/tweetstream/client.rb:525:in `invoke_callback'
    /var/lib/gems/1.8/gems/tweetstream-2.3.0/lib/tweetstream/client.rb:533:in `yield_message_to'
    /var/lib/gems/1.8/gems/tweetstream-2.3.0/lib/tweetstream/client.rb:471:in `respond_to'
    /var/lib/gems/1.8/gems/tweetstream-2.3.0/lib/tweetstream/client.rb:411:in `connect'
    /var/lib/gems/1.8/gems/em-twitter-0.2.1/lib/em-twitter/connection.rb:296:in `call'
    /var/lib/gems/1.8/gems/em-twitter-0.2.1/lib/em-twitter/connection.rb:296:in `invoke_callback'
    /var/lib/gems/1.8/gems/em-twitter-0.2.1/lib/em-twitter/connection.rb:143:in `handle_stream'
    /var/lib/gems/1.8/gems/em-twitter-0.2.1/lib/em-twitter/connection.rb:193:in `on_body'
    /var/lib/gems/1.8/gems/em-twitter-0.2.1/lib/em-twitter/connection.rb:192:in `each'
    /var/lib/gems/1.8/gems/em-twitter-0.2.1/lib/em-twitter/connection.rb:192:in `on_body'
    /var/lib/gems/1.8/gems/em-twitter-0.2.1/lib/em-twitter/connection.rb:74:in `<<'
    /var/lib/gems/1.8/gems/em-twitter-0.2.1/lib/em-twitter/connection.rb:74:in `receive_data'
    /var/lib/gems/1.8/gems/eventmachine-1.0.0/lib/eventmachine.rb:187:in `run_machine'
    /var/lib/gems/1.8/gems/eventmachine-1.0.0/lib/eventmachine.rb:187:in `run'
    /var/lib/gems/1.8/gems/tweetstream-2.3.0/lib/tweetstream/client.rb:385:in `start'
    /var/lib/gems/1.8/gems/tweetstream-2.3.0/lib/tweetstream/client.rb:128:in `filter'
    /var/lib/gems/1.8/gems/tweetstream-2.3.0/lib/tweetstream/client.rb:106:in `follow'
    tracker.rb:27

You can't just store a Twitter object. You have to turn that object into a YAML or JSON hash first. — Linuxios, Nov 13 '12 at 13:57

jimoleary · Answer 1 · 2013-05-21T07:17:48.183

If you want to blindly dump everything into MongoDB then you can just do a :

client.follow(<user_id>,<user_id>) do |status|
  data = status.to_hash
  id = tweets.insert(data)
end

If you want to be a bit more selective then you can try something like :

# only add the following fields to the database
ONLY = %w{created_at text geo coordinate id_str}

client.follow(<user_id>,<user_id>) do |status|
  data = status.to_hash.select{|k,v| ONLY.include?(k.to_s)}
  id = tweets.insert(data)
end

Or :

# add everything except the following fields to the database
EXCEPT = %w{entities}

client.follow(<user_id>,<user_id>) do |status|
  data = status.to_hash.reject{|k,v| EXCEPT.include?(k.to_s)}
  id = tweets.insert(data)
end

score 0 · Answer 2 · answered Nov 13 '12 at 15:10

0

It seems looking at the docs that Client.follow just returns a generic ruby object. You have to take the relevant fields from the object and pull them out into json (or, more advisable, a mongoid object) to send to the database.

answered Nov 13 '12 at 15:10

shelman

2,689
15
17

Thanks for the answer. I am pulling out specific fields using status. for e.g status.text, status.geo. Most of them work fine but status.coordinates, status.contributors fail with undefinedmethod error. Any idea why? – user1820971 Nov 17 '12 at 01:06
Most of tweets fields are coming fine but for entities, I am getting this error BSON::InvalidDocument Cannot serialize an object of class Twitter:entity::User_Mention into BSON – user1820971 Nov 18 '12 at 18:23
I'm not sure, sorry... have you tried inspecting them with a debugger to see what status.coordinates and status.contributors actually are? – shelman Nov 19 '12 at 15:40

score 0 · Answer 3 · answered Mar 11 '13 at 20:35

has_key? is a method available in the Hash superclass. What it's trying to do is make a json string to pass to Mongo. Except that you are passing a string, has_key? is not part of the String class. Just convert it to a hash and you should be good to go.

client.follow(<user_id>,<user_id>) do |status|
    id = tweets.insert({:status => status.text}, :safe => true)
end

Storing tweets using tweetstream in MongoDB fails with following error

3 Answers3