Mechanize Traverse & Parse

Question

I'm trying to walk through a hitlist of pages found by Mechanize. Upon a search, which is working just fine, I get a hitlist with 10 records per page. A bottom navigation system takes me to the record count > 10. Display 10 per page. So 53 records = 6 "group pages" as I'm calling them.

What I want to do is use the top search results page to do the following:

1. Grab the html behind every record on this list. I can do that through an iteration.

2. Follow through the '[Next]' link at the bottom and repeat both 1 and 2 until there is no more 2. Essentially this will get every record.

I'm having an issue with moving off the first page into the second page. I'm grabbing the html behind the first 10 records, but then the system bonks on me. Not sure why. I thought I was iterating through the group pages, but it isn't advancing past the first group page.

    counter = 1
    puts "Counter: #{counter}"

    while agent.page.links_with(:text => '[Next]').count == 1
        page = agent.page.link_with(:text => '[Next]').click
        puts "Counter/Next: #{counter} / #{agent.page.links_with(:text => '[Next]').count}" 
        agent.page.links_with(text: '[complete profile]').each do |link|
            a = link.click      # link.click goes to each company page 
            r = a.body.to_s 
            r = r.gsub(/^\s+\n/, "")

            # enter company into db
            class Company < ActiveRecord::Base
            end

            company = Company.new
            company.cdate = DateTime.now
            company.status = 'new'
            company.requestID = req_id
            company.html = r
            company.save    
        end
        counter += 1
    end

Any insight appreciated. I know I'm close.

Cheers

Answer 1

Solution: move the click method behind page:

page = agent.click(page.link_with(:text => "[Next]"))