Ruby EOFError with open-uri and loop

Question

I'm attempting to build a web crawler and ran into a bit of a snag. Basically what I'm doing is extracting the links from a web page and pushing each link to a queue. Whenever the Ruby interpreter hits this section of code:

links.each do |link|
  url_frontier.push(link)
end

I receive the following error:

/home/blah/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/net/protocol.rb:141:in `read_nonblock': end of file reached (EOFError)

If I comment out the above block of code I get no errors. Please, any help would be appreciated. Here is the rest of the code:

require 'open-uri'
require 'net/http'
require 'uri'

class WebCrawler
  def self.Spider(root)
    eNDCHARS = %{.,'?!:;}
    num_documents = 0
    token_list = []
    url_repository = Hash.new
    url_frontier = Queue.new

    url_frontier.push(root.to_s)
    while !url_frontier.empty? && num_documents < 10
    url = url_frontier.pop
      if !url_repository.has_key?(url)
        document = open(url)
        html = document.read

        # extract url's
        links = URI.extract(html, ['http']).collect { |u| eNDCHARS.index(u[-1]) ? u.chop : u }

        links.each do |link|
          url_frontier.push(link)
        end

        # tokenize
        Tokenizer.tokenize(document).each do |word|
          token_list.push(IndexStructures::Term.new(word, url))
        end

        # add to the repository
        url_repository[url] = true
        num_documents += 1
      end
    end

    # sort by term (primary) and document id (secondary) in reverse to aid in the    construction of the inverted index
    return num_documents, token_list.sort_by! { |term| [term.term, term.document_id]}.reverse!
  end
end

score 0 · Answer 1 · answered Oct 07 '12 at 20:01

I encountered the same error but with Watir-webdriver, running firefox in headless mode. What I found out was, if I was running two of my applications in parallel and I destroy "headless" in one of the applications, it automatically kills the other one as well with the exact error you quoted. Though my situation is not the same as yours, I think the issue is related to prematurely closing the file handle externally while your application is still using it. I removed the destroy command from my application and the error disappeared.

Hope this helps.

Ruby EOFError with open-uri and loop

1 Answers1