2

I'm using RoR, I will specify a link to a web page in my application and here are the things that I want to do

(1) I want to extract all the links in the web page

(2) Find if they are links to pdf file(basically a pattern match)

(3)I want to download file in link(a pdf for example) and store them on my system.

I tried using Anemone, but it crawls the entire website which overshoots my needs and also how do I download the files in corresponding links?

Cheers

theReverseFlick
  • 5,894
  • 8
  • 32
  • 33

2 Answers2

10

Have a look at Nokogiri aswell.

require 'nokogiri'
require 'open-uri'
doc = Nokogiri::HTML(open('http://www.thatwebsite.com/downloads'))

doc.css('a').each do |link|
  if link['href'] =~ /\b.+.pdf/
    begin
      File.open('filename_to_save_to.pdf', 'wb') do |file|
        downloaded_file = open(link['href'])
        file.write(downloaded_file.read())
      end
    rescue => ex
      puts "Something went wrong...."
    end
  end
end

You might want to do some better exception catching, but I think you get the idea :)

simonwh
  • 1,017
  • 8
  • 21
1

Have you tried scrapi? You can scrape the page with css selectors.

Ryan Bates also made a screencast about it.

To download the files you can use open-uri

require 'open-uri'  
url = "http://example.com/document.pdf"
file = open(url)  
c = file.read()
Tarscher
  • 1,923
  • 1
  • 22
  • 45
  • But I have trouble using 'scrapi', I'm using ruby 1.8.7. It says Scraper::Reader::HTMLParseError: Unable to load /Library/Ruby/Gems/1.8/gems/scrapi-1.2.0/lib/tidy/libtidy.dylib – theReverseFlick Feb 04 '11 at 12:41