3

Is there a better way to extract the visible text on a web page using Nokogiri? Currently I use the inner_text method, however that method counts a lot of JavaScript as visible text. The only text I want to capture is the visible text on the screen.

For example, in IRB if I do the following in Ruby 1.9.2-p290:

require 'nokogiri'
require 'open-uri'
doc = Nokogiri::HTML(open("http://www.bodybuilding.com/store/catalog/new-products.jsp?addFacet=REF_BRAND:BRAND_MET_RX"))
words = doc.inner_text
words.scan(/\w+/)

If I search for the word "function" I see that it appears 20 times in the list, however if I go to http://www.bodybuilding.com/store/catalog/new-products.jsp?addFacet=REF_BRAND:BRAND_MET_RX the word "function" does not appear anywhere in the visible text.

Can I ignore JavaScript or is there a better way of doing this?

the Tin Man
  • 158,662
  • 42
  • 215
  • 303
akaDanPaul
  • 283
  • 3
  • 9

3 Answers3

7

You could try:

require 'nokogiri'
require 'open-uri'

doc = Nokogiri::HTML(open("http://www.bodybuilding.com/store/catalog/new-products.jsp?addFacet=REF_BRAND:BRAND_MET_RX"))

doc.traverse{ |x|
    if x.text? && x.text !~ /^\s*$/
        puts x.text
    end
}

I have not done much with Nokogiri, but I believe this should find/output all text nodes in the document that are not blanks. This at least seems to be ignoring the javascript and all the text I checked was visible on the page (though some of it in the dropdown menus).

Community
  • 1
  • 1
Justin Ko
  • 46,526
  • 5
  • 91
  • 101
  • this works but you should avoid `and` in a conditional because it is meant for control flow and has a different order of operations than the intended `&&` operator. Also, there is a `!~` operator you can use instead of `not =~` – Ben Jun 02 '17 at 21:51
2

You can ignore JavaScript and there is a better way. You're ignoring the power of Nokogiri. Badly.

Rather than provide you with the direct answer, it will do you well to learn to "fish" using Nokogiri.

In a document like:

<html>
  <body>
    <p>foo</p>
    <p>bar</p>
  </body>
</html>

I recommend starting with CSS accessors because they're generally more familiar to people:

  • doc = Nokogiri::HTML(var_containing_html) will parse and return the HTML DOM in doc.
  • doc.at('p') will return a Node, which basically points to the first <p> node.
  • doc.search('p') will return a NodeSet of all matching nodes, which acts like an array, in this case all <p> nodes.
  • doc.at('p').text will return the text inside a node.
  • doc.search('p').map{ |n| n.text } will return all the text in the <p> nodes as an array of text strings.

As your document gets more complex you need to drill down. Sometimes you can do it using a CSS accessor, such as 'body p' or something similar, and sometimes you need to use XPaths. I won't go into those but there are great tutorials and references out there.

Nokogiri's tutorials are very good. Go through them and they will reveal all you need to know.

In addition, there are many answers on Stack Overflow discussing this sort of problem. Check out the "Related" links on the right of the page.

the Tin Man
  • 158,662
  • 42
  • 215
  • 303
  • Good answer, the map function to array will definitely be useful for me later again – abcde123483 Mar 29 '13 at 13:10
  • 2
    This answer is off topic, the question was to grab visible inner-text of the entire page, not of a bunch of elements. Even using `body`, the OP will not be able to get rid of the `javascript` tag. – sidney Oct 20 '14 at 09:47
  • Off topic? Hardly. The OP was not using Nokogiri correctly, and, instead of trying to find the nodes that were needed, was trying to do a blanket grab of all text and ended up with JavaScript which IS text. Instead, by selectively cherry-picking the nodes desired, those that actually displayed, the text on the screen could be extracted without any garbage. The OP wanted a "better way to do this", and that involves understanding how to correctly extract data, which is what this answer provides. Your comment only proves my point. – the Tin Man Oct 20 '14 at 13:57
  • 1
    Capybara seems to have a method which does exactly what the OP wants: @session.text ... so I don't think the question's crazy. In fact, I'm pretty sure capybara is based on nokogiri so it must be possible if that's true. – Carpela Sep 09 '15 at 13:00
0

Ignore the tags where JavaScript lives (<script>). While we’re at it, we should also ignore CSS (<styles>).

require 'nokogiri'
require 'open-uri'

doc = Nokogiri::HTML(URI.open("http://www.bodybuilding.com/store/catalog/new-products.jsp?addFacet=REF_BRAND:BRAND_MET_RX"))
doc.css('style').each(&:remove)
doc.css('script').each(&:remove)

puts doc.text

# Alternatively, for cleaner output:
# puts doc.text.split("\n").map(&:strip).reject(&:empty?)
user137369
  • 5,219
  • 5
  • 31
  • 54