1

Thanks for your time.

I'm working up a Ruby script to parse a CSV of urls and evaluate them on a variety of dimensions to see if certain tags and attributes are present or confirm to certain pattern.

I'm using Nokogiri, open-uri and the patch for open-url to allow the script to follow redirections, open_uri_redirections.

On a handful of problematic domains, I encounter an error and the script encounters a runtime error:

Loading https://www.exampleproblemdomain.com C:/Ruby24-x64/lib/ruby/2.4.0/open-uri.rb:233:in `open_loop': HTTP redirection loop: https://www.exampleproblemdomain.com (RuntimeError)
        from C:/Ruby24-x64/lib/ruby/2.4.0/open-uri.rb:151:in `open_uri'
        from C:/Ruby24-x64/lib/ruby/gems/2.4.0/gems/open_uri_redirections-0.2.1/lib/open-uri/redirections_patch.rb:55:in `open_uri'
        from C:/Ruby24-x64/lib/ruby/2.4.0/open-uri.rb:721:in `open'
        from C:/Ruby24-x64/lib/ruby/2.4.0/open-uri.rb:35:in `open'
        from arraycsv.rb:16:in `block in <main>'
        from C:/Ruby24-x64/lib/ruby/2.4.0/csv.rb:866:in `each'
        from C:/Ruby24-x64/lib/ruby/2.4.0/csv.rb:866:in `each'
        from arraycsv.rb:14:in `<main>'

The open_uri_redirections gem works well for most domains but a select few encounter this issue and I can't figure out why, even when I use a redirection extension like Ayima to follow the chain of redirections.

Could it be a 302 vs. a 301 creating the issue?

user2308493
  • 11
  • 1
  • I've found that open-uri fails if the website is setting a cookie and then redirecting after the cookie is set because open-uri is not saving the cookie (as far as I understand it). You can test this by blocking cookies temporarily in your browser and attempting to load the url again. – SWoo Feb 27 '19 at 06:12

0 Answers0