How do I ignore the nil values in the loop with parsed values from Mechanize?

Question

In my text file are a list of URLs. Using Mechanize I'm using that list to parse out the title and meta description. However, some of those URL pages don't have a meta description which stops my script with a nil error:

undefined method `[]' for nil:NilClass (NoMethodError)

I've read up and seen solutions if I were using Rails, but for Ruby I've only seen reject and compact as possible solutions to ignore nil values. I added compact at the end of the loop, but that doesn't seem to do anything.

require 'rubygems'
require 'mechanize'

File.readlines('parsethis.txt').each do |line|
    page = Mechanize.new.get(line)
    title = page.title
    metadesc = page.at("head meta[name='description']")[:content]
    puts "%s, %s, %s" % [line.chomp, title, metadesc] 
end.compact!

It's just a list of urls in a text like this:

http://www.a.com
http://www.b.com

This is what will output in the console for example:

 http://www.a.com, Title, This is a description.

If within the list of URLs there is no description or title on that particular page, it throws up the nil error. I don't want it to skip any urls, I want it to go through the whole list.

Answer 1

Here is one way to do it:

Edit( for added requirement to not skip any url's):

metadesc = page.at("head meta[name='description']")
puts "%s, %s, %s" % [line.chomp, title, metadesc ? metadesc[:content] : "N/A"]

Answer 2

This is untested but I'd do something like this:

require 'open-uri'
require 'nokogiri'

page_info = {}
File.foreach('parsethis.txt') { |url|
  page = Nokogiri::HTML(open(url))
  title = page.title
  meta_desc = page.at("head meta[name='description']")
  meta_desc_content = meta_desc ? meta_desc[:content] : nil
  page_info[url] = {:title => title, :meta_desc => meta_desc_content} 
}

page_info.each do |url, info|
  puts [
    url,
    info[:title],
    info[:meta_desc]
  ].join(', ')
end

• File.foreach iteratively reads a file, returning each line individually.
• page.title could return a nil if a page doesn't have a title; titles are optional in pages.

• I break down accessing the meta-description into two steps. Meta tags are optional in HTML so they might not exist, at which point a nil would be returned. Trying to access a content= parameter would result in an exception. I think that's what you're seeing.

  Instead, in my code, meta_desc_content is conditionally assigned a value if the meta-description tag was found, or nil.

The code populates the page_info hash with key/value pairs of the URL and its associated title and meta-description. I did it this way because a hash-of-hashes, or possibly an array-of-hashes, is a very convenient structure for all sorts of secondary manipulations, such as returning the information as JSON or inserting into a database.

As a second step the code iterates over that hash, retrieving each key/value pair. It then joins the values into a string and prints them.

There are lots of things in your code that are either wrong, or not how I'd do them:

• File.readlines('parsethis.txt').each returns an array which you then have to iterate over. That isn't scalable, nor is it efficient. File.foreach is faster than File.readlines(...).each so get in the habit of using it unless you are absolutely sure you know why you should use readlines.
• You use Mechanize for something that Nokogiri and OpenURI can do faster. Mechanize is a great tool if you are working with forms and need to navigate a site, but you're not doing that, so instead you're dragging around additional code-weight that isn't necessary. Don't do that; It leads to slow programs among other things.
• page.at("head meta[name='description']")[:content] is an exception in waiting. As I said above, meta-descriptions are not necessarily going to exist in a page. If it doesn't then you're trying to do nil[:content] which will definitely raise an exception. Instead, work your way down to the data you want so you can make sure that the meta-description exists before you try to get at its content.
• You can't use compact or compact! the way you were. An each block doesn't return an array, which is the class you need for compact or compact!. You could have used map but the logic would have been messy and puts inside map is rarely used. (Probably shouldn't be used is more likely but that's a different subject.)