Automated website interaction - Mechanzie - Rails

Question

I'm using the Mechanize gem to automate interaction with a website form.

The site i'm trying to interact with is http://www.tastekid.com/like/books

I'm trying to automatically submit a string to query in the form and return the suggested books in an array.

Following the guide, i've pretty printed the page layout to find the form name, but, I am just finding a form with no name, nill:

require 'rubygems'
require 'mechanize'

agent = Mechanize.new
page = agent.get('http://www.tastekid.com/like/books')
pp page

How do I enter a string, submit the form and return the results in the form of an array?

aceofbassgreg · Answer 1 · 2014-09-30T16:36:03.067

1

Following the guide, you can get the form:

form = page.form

I didn't see a name on the form, and I actually got two forms back: one on the page and one hidden.

I called

form.fields.first.methods.sort  #not the hidden form

and saw that I could call value on the form, so I set it as such:

form.fields.first.value = "Blood Meridian"

then I submitted and pretty printed:

page = agent.submit(form)

This should work for you!

edited Sep 30 '14 at 16:36

answered Sep 30 '14 at 16:13

aceofbassgreg

3,837
2
32
42

Thank you! How do I grab the book titles out of the # object? – Katie H Sep 30 '14 at 16:20
When doing a pretty print, it looks like all the titles are under Mechanize::Page::Link, how do I select just the book titles so I can put them in an array? – Katie H Sep 30 '14 at 16:29
1

You will want to read up on `Nokogiri` to figure out how to parse the contents of the search results page. – aceofbassgreg Sep 30 '14 at 16:51

orde · Answer 2 · 2014-09-30T17:00:33.050

1

You could use the form_with method to locate the form you want. For example:

require 'mechanize'

agent = Mechanize.new
page = agent.get('http://www.tastekid.com/like/books')

the_form_you_want = page.form_with(:id => "searchFrm")  # form_with
the_form_you_want.q = 'No Country for Old Men'
page = agent.submit(the_form_you_want)
pp page

It looks like the book titles all have the same class attribute. To extract the book titles, use the links_with method and pass in the class as a locator:

arr = []

page.links_with(:class => "rsrc").each do |link|
  arr << link.text
end

But @aceofbassgreg is right. You'll need to read up on the mechanize and nokogiri documentation...

edited Sep 30 '14 at 17:00

answered Sep 30 '14 at 16:29

orde

5,233
6
31
33

Is this way more efficient than the other answer? How do I extract the book titles only and push them in an array? The pretty print gives me a bunch of stuff. – Katie H Sep 30 '14 at 16:34
Cormac McCarthy for the win on Stack Overflow! – aceofbassgreg Sep 30 '14 at 19:26
@aceofbassgreg: Indeed! :) – orde Sep 30 '14 at 23:40

score 1 · Accepted Answer · answered Oct 01 '14 at 08:35

These answers feel a little cluttered to me, so let me try to make it simpler:

page = agent.get 'http://www.tastekid.com/like/books'

there's only one form, so:

form = page.form
form['q'] = 'twilight'

submit the form

page = form.submit

print the text from the a's

puts page.search('.books a').map &:text

Automated website interaction - Mechanzie - Rails

3 Answers3