rails too many open files using Feedzirra

Question

I am getting a LoadError - "Too many open files" when using Feedzirra. I am running it on my development server using the default WEBrick server.

I am parsing only 2 feeds. What is the problem?

Any progress on this? If not, please describe your environment and post relevant messages and code. I just received a hit on this using Windows 8, Ruby 193p4448 and Rails 4.0.0 with Feedzirra. However, I decreased my number of feeds to 2 and it worked. To me, that would mean that I need to fix either Windows and/or the Thin webserver I am using. Or, perhaps, migrate to Apache as I have planned to do at some point. — Richard_G, Aug 28 '13 at 02:00

score 0 · Answer 1 · answered Jan 11 '14 at 22:58

I had the same issue with Feedzirra. You can notice that it leaves TCP connections in CLOSE_WAIT state forever, hence causing the problem.

It appears to be curb gem specific that is used to fetch feeds. Another project depending on libcurl had the same issue. They have fixed it by setting 'CURLOPT_FORBID_REUSE' option.

I've tried to do the same for Feedzirra but didn't succeed. Even with this option I had a growing number of CLOSE_WAIT sessions and Too many open files error eventually.

So I did the most straightforward thing, I download feeds using Net::HTTP:

def get_contents(furl)
  url = URI.parse(furl)
  req = Net::HTTP::Get.new(url.to_s)

  res = Net::HTTP.start(url.host, url.port) { |http|
    http.request(req)
  }

  unless res.kind_of? Net::HTTPSuccess
    puts "can't get feed #{url.to_s}: #{res.code}"
    return nil
  end

  res.body
end

Then I parse the XML with Feedzirra:

xml = get_contents(furl)
feedin = Feedzirra::Feed.parse xml

No more stuck connections and no more errors. You may also want to add better error handling to this sample code.

rails too many open files using Feedzirra

1 Answers1