Submitting a form via an element with Ruby Mechanize

Question

I've got a set of pages that I'm trying to scrape with Mechanize in Ruby. On some of the pages, they redirect to a page that wants me to fill out a select-list form and then submit it. The problem is with the button that submits the form, which is an <a> element. It looks like this:

<a class="css_class" href="#" onclick="RunSomeScript; return false;">
  <span>
    Enter
  </span>
</a>

The magic seems to be happening with the RunSomeScript script; it is the thing that seems to bypass the redirect page and take me to the page with the data that I'm trying to scrape. Unfortunately, I can't seem to figure out how to properly submit the form with Mechanize. I tried using the Mechanize#click method on the <a> element by passing in the <a>'s href attribute to it, but that hasn't seemed to work either. How do I automate a click on this link (i.e., a submission of the form) properly and/or run the RunSomeScript script in order to submit the form on this redirect page?

score 0 · Answer 1 · answered May 08 '15 at 04:33

0

I don't if or how you can get mechanize to support javascript, but I see other projects that you might look at instead:

Capybara, using the poltergeist (phantomjs) driver or watir, perhaps also using its phantomjs support.

answered May 08 '15 at 04:33

ysth

96,171
6
121
214

score 0 · Accepted Answer · answered May 12 '15 at 07:04

I think you can directly find the form and submit with Mechanize i.e

# if condition checks if the form exists with the id the add some fields to it and then submit    
if page.search("#ctl00_ContentPlaceHolder1_ctrlResults_gvResults_ctl01_lbNext").count > 0
              form = page.forms.first
              form.add_field! "__EVENTTARGET", "ctl00$ContentPlaceHolder1$ctrlResults$gvResults$ctl01$lbNext"
              form.add_field! "__EVENTARGUMENT", ""
              page = form.submit

In your case you can find the form using some id or just use

form = page.forms.first  
# do whatever you want with this form and then 
page = form.submit

Thanks, this works for me! Also, sorry for taking such a long time to accept the answer; I kind of had some things come up at the time that I was working on this and dropped it for some months. — GDP2, Nov 24 '15 at 07:34

Submitting a form via an element with Ruby Mechanize

2 Answers2