Skip large pages in Mechanize (Ruby)

Question

I'm trying to skip processing a few large pages (some over 10MB) scattered in a result set, as Mechanize (version 2.7.3) crawls an array of links.

Unfortunately I can't find a 'content-length' property or a similar indicator. The Mechanize::FileResponse class has a content_length method but Mechanize::Page does not.

Current approach

At the moment I'm calling content.length on the page. This is very slow when one of the large pages is crawled:

detail_links.each do |detail_link|

  detail_page = detail_link.click

  # skip long pages
  break if detail_page.content.length > 100_000

  # rest of the processing
end

Content_length during response_read:

In the Mechanize source code I found a reference to content_length when the response is read. Is querying the response properties a possible solution?

# agent.rb extract from the Mechanize project
def response_read response, request, uri
    content_length = response.content_length

    if use_tempfile? content_length then
      body_io = make_tempfile 'mechanize-raw'
    else
      body_io = StringIO.new.set_encoding(Encoding::BINARY)
    end

score 1 · Accepted Answer · edited May 23 '17 at 11:57

Mechanize will normally "get" the entire page. Instead you should use a head request first to get the page size, then conditionally get the page. See "How can I perform a Head request using mechanize in Ruby" for an example.

The thing to be careful of is that a dynamically generated resource might not have a known size when you do the head request, so you could get a response without the size entry. Notice that in the selected answer for the question linked above, that Google didn't return the content-length header because it's a dynamically generated page. Static pages and resources should have the header... unless the server doesn't return them for some reason.

The Mechanize documentation mentions this:

Problems with content-length

Some sites return an incorrect content-length value. Unlike a browser, mechanize raises an error when the content-length header does not match the response length since it does not know if there was a connection problem or if the mismatch is a server bug.

The error raised, Mechanize::ResponseReadError, can be converted to a parsed Page, File, etc. depending upon the content-type:

agent = Mechanize.new
uri = URI 'http://example/invalid_content_length'

begin
  page = agent.get uri
rescue Mechanize::ResponseReadError => e
  page = e.force_parse
end

In other words, while head can help, it's not necessarily going to give you enough information to allow you to skip huge pages. You have to investigate the site you're crawling and learn how their server responds.

Thanks for the informative answer - I didn't know about ```head```. Looking at the response header I can see there's no ```content-length``` from the server (it is indeed a dynamically generated page). — odlp, Jul 31 '14 at 08:53
Well, if the server fails to send the header just for those pages you want to skip, then you can key off that and ignore any without the `content-length` field. But, from experience, I know you can't count on that behavior with every HTTPd and site you encounter. — the Tin Man, Jul 31 '14 at 15:24

Skip large pages in Mechanize (Ruby)

Current approach

Content_length during response_read:

1 Answers1