I've read that Nokogiri/Mechanize (Ruby) for example are not good at traversing multiple pages, but may be better with sites that use Ajax.
The sites I want to scrape are multi-page forms, with some ajax overlays. Speed is important. These sites all display prices, so I am making a price aggregator.
I use Capybara with Webkit to a headless browser.
You'll need install capybara gem, and webkit gem as well.
https://github.com/thoughtbot/capybara-webkit
The syntax is very simple.
agent.visit 'some url'
agent.execute_script 'javascript here'
The gem also have page management, or you may simply go back to previous page by execute a javascript going back.
ag.execute_script("window.history.go(-1)")
