I'm trying to scrape concert data from a bunch of different websites. I've written several scripts, each of which scrapes data from a particular website and organizes it into a hash with a predictable structure. Each of these scripts has the function scrape
to accomplish this. I have all of these scraper scripts in a directory. I want to then write a master script which, for each script in the directory, calls that script's scrape
and adds the data into the database.
I plan on making this master script a Resque worker such that the site will scrape each site in the background daily.
How do I accomplish this in the master script? Right now I go through the directory like so:
Dir.glob(/app/workers/scraped_venues/*.rb) do |venue_scraper|
# call that script's `scrape` function
# use data from that `scrape` call
end