Having problems with Ruby file from Dashing

Question

I am having trouble with twitter_user.rb, which is supposed to get the number of tweets, followers, and following of a given Twitter username.

I assume that I am supposed to replace TWITTER_USERNAME in line 9 with the Twitter username that I am interested in. I did that and started dashing but I got:

scheduler caught exception:
undefined method '[]' for nil:NilClass
/.../jobs/twitter_user.rb:19:in 'block in <top (required)>'

It looks like the problem is with line 19 which is:

tweets = /profile["']>[\n\t\s]*<strong>([\d.,]+)/.match(response.body)[1].delete('.,').to_i

Can anybody tell me what is going on and how to fix it?

The problem is the code wasn't written in a smart manner. Using regular expressions to parse XML or HTML is a wreck-in-waiting unless it's content OWNED by the person writing the code and that content won't change and it's a trivial task. Change one of those three and the odds of the code breaking goes up quickly. We need to see a sample of the HTML you're encountering so we can reproduce the problem. "Questions concerning problems with code you've written must describe the specific problem — and include valid code to reproduce it — in the question itself. See http://SSCCE.org for guidance." — the Tin Man, Nov 11 '13 at 16:34
I think you are right. It does not even look like the job is running on the webapp that they made http://foobugs-dashboard.herokuapp.com/tv — user1893354, Nov 11 '13 at 16:41

score 1 · Accepted Answer · answered Nov 11 '13 at 17:48

Your assumption is incorrect. The program is looking for an environment variable called TWITTER_USERNAME that is set to the relevant user name. If that variable doesn't exist then the code uses foobugs instead.

If you would rather modify the code than set up an environment variable, then change

twitter_username = ENV['TWITTER_USERNAME'] || 'foobugs'

to

twitter_username = 'myusername'

the Tin Man · Answer 2 · 2013-11-11T16:55:41.323

This is untested code, but it's a general idea how it should have been written. If you clone the source on the original page you can adjust it for your own purposes (i.e. fix it):

require 'nokogiri'

doc = Nokogiri::XML(content)

tweets = doc.at('profile strong').text.delete('.,').to_i
following = doc.at('following strong').text.delete('.,').to_i
followers = doc.at('followers strong').text.delete('.,').to_i

The above three lines can be reduced to something like:

tweets, following, followers = %w[profile following followers].map{ |tag|
  doc.at("#{ tag } strong").text.delete(',.').to_i
}

Again, without a usable sample of the XML/HTML I can't do much more, but as a practice we (programmers) shouldn't use regular expressions to try to parse XML or HTML. It's much to easy to break a pattern with either of those types of files.

score 0 · Answer 3 · answered Nov 15 '13 at 08:14

I managed to solve the same issue for myself by using the twitter API instead to pull out the relevant information. It seems the web page had changed too much for the scraping to work and it could also stop working again at no notice as various people have already said...

This is the solution I used.

#### Get your twitter keys & secrets:
#### https://dev.twitter.com/docs/auth/tokens-devtwittercom
Twitter.configure do |config|
  config.consumer_key = 'YOUR_CONSUMER_KEY'
  config.consumer_secret = 'YOUR_CONSUMER_SECRET'
  config.oauth_token = 'YOUR_OAUTH_TOKEN'
  config.oauth_token_secret = 'YOUR_OAUTH_SECRET'

end

twitter_username = 'foobugs'

MAX_USER_ATTEMPTS = 10
user_attempts = 0

SCHEDULER.every '10m', :first_in => 0 do |job|
  begin
    tw_user = Twitter.user("#{twitter_username}")
    if tw_user
        tweets = tw_user.statuses_count
        followers = tw_user.followers_count
        following = tw_user.friends_count

        send_event('twitter_user_tweets', current: tweets)
        send_event('twitter_user_followers', current: followers)
        send_event('twitter_user_following', current: following)

    end
  rescue Twitter::Error => e
    user_attempts = user_attempts +1
    puts "Twitter error #{e}"
    puts "\e[33mFor the twitter_user widget to work, you need to put in your twitter API keys in the jobs/twitter_user.rb file.\e[0m"
    sleep 5
    retry if(user_attempts < MAX_USER_ATTEMPTS)
  end
end

score 0 · Answer 4 · answered Mar 03 '14 at 13:03

I have resolved by substituting this line:

followers = /<strong>([\d.]+)<\/strong> Follower/.match(response.body)[0].delete('.,').to_i

with these two:

followers_count_metadata = /followers_count&quot;:[\d]+/.match(response.body)
followers = /[\d]+/.match(followers_count_metadata.to_s).to_s

Having problems with Ruby file from Dashing

4 Answers4

Linked