How to use Mechanize to parse local file

Question

I'm using Ruby and Mechanize to parse a local HTML file but I can't do it. This works if I use a URL though:

agent = Mechanize.new
#THIS WORKS
#url = 'http://www.sample.com/sample.htm'
#page = agent.get(url) #this seems to work just fine but the following below doesn't

#THIS FAILS
file = File.read('/home/user/files/sample.htm') #this is a regular html file
page = Nokogiri::HTML(file)
pp page.body #errors here

page.search('/div[@class="product_name"]').each do |node|
  text = node.text  
  puts "product name: " + text.to_s
end

The error is:

/home/user/code/myapp/app/models/program.rb:35:in `main': undefined method `body' for #<Nokogiri::HTML::Document:0x000000011552b0> (NoMethodError)

How do I get a page object so that I can search on it?

The error is correct, Nokogiri's Document doesn't have a "`body`" method. Nokogiri is the next layer down below Mechanize so you have to use its methods. — the Tin Man, Feb 07 '20 at 18:08

score 1 · Accepted Answer · edited Oct 07 '21 at 11:07

Mechanize uses URI strings to point to what it's supposed to parse. Normally we'd use a "http" or "https" scheme to point to a web-server, and that's where Mechanize's strengths are, but other schemes are available, including "file", which can be used to load a local file.

I have a little HTML file on my Desktop called "test.rb":

<!DOCTYPE html>
<html>
<head></head>
<body>
<p>
Hello World!
</p>
</body>
</html>

Running this code:

require 'mechanize'

agent = Mechanize.new
page = agent.get('file:/Users/ttm/Desktop/test.html')
puts page.body

Outputs:

<!DOCTYPE html>
<html>
<head></head>
<body>
<p>
Hello World!
</p>
</body>
</html>

Which tells me Mechanize loaded the file, parsed it, then accessed the body.

However, unless you need to actually manipulate forms and/or navigate pages, then Mechanize is probably NOT what you want to use. Instead Nokogiri, which is under Mechanize, is a better choice for parsing, extracting data or manipulating the markup and it's agnostic as to what scheme was used or where the file is actually located:

require 'nokogiri'

doc = Nokogiri::HTML(File.read('/Users/ttm/Desktop/test.html'))
puts doc.to_html

which then output the same file after parsing it.

Back to your question, how to find the node only using Nokogiri:

Changing test.html to:

<!DOCTYPE html>
<html>
<head></head>
<body>
<div class="product_name">Hello World!</div>
</body>
</html>

and running:

require 'nokogiri'

doc = Nokogiri::HTML(File.read('/Users/ttm/Desktop/test.html'))
doc.search('div.product_name').map(&:text)
# => ["Hello World!"]

shows that Nokogiri found the node and returned the text.

This code in your sample could be better:

text = node.text  
puts "product name: " + text.to_s

node.text returns a string:

doc = Nokogiri::HTML('<p>hello world!</p>')
doc.at('p').text # => "hello world!"
doc.at('p').text.class # => String

So text.to_s is redundant. Simply use text.

You are a blessing unto the world. Thank you so much. And for teaching me how to be more elegant with my code. — heebee313, Feb 08 '20 at 00:05
I'm not sure about the blessing, but it's just what we're supposed to do here. Nokogiri is a really cool tool; I used it to write a big RSS/RDF/Atom aggregator for a company and found it very easy to use, especially compared to other, earlier, tools we'd had to work with. — the Tin Man, Feb 08 '20 at 00:17

How to use Mechanize to parse local file

1 Answers1