Mechanize::ResponseReadError - Content-Length does not match response body length

Question

I have encountered this error while requesting a page with Mechanize:

Mechanize::ResponseReadError
Content-Length (17317) does not match response body length (17070) (Mechanize::ResponseReadError)

Any thoughts on why this occurs and how I could get about fixing it are much appreciated!

Can you intercept the response and check to see whether the response body is indeed shorter than the Content-Length? — Slicedpan, May 28 '14 at 13:09

score 1 · Accepted Answer · answered May 28 '14 at 13:27

It happens that sites return a wrong content length-value. Catch the error and force page parsing.

agent = Mechanize.new
begin
  page = agent.get 'http://bad.com'
rescue Mechanize::ResponseReadError => e
  page = e.force_parse
end

You can also set agent.ignore_bad_chunking to true — but then beware of possible silent content loss.

score 0 · Answer 2 · answered May 28 '14 at 13:37

It occurs because Content-Length header is not equal to the size of response-body length.

Check the below specs taken by mechanize gem. it will raise same error.

  def test_response_read_content_length_mismatch
    def @res.content_length() 5 end
    def @res.read_body() yield 'part' end

    e = assert_raises Mechanize::ResponseReadError do
      @agent.response_read @res, @req, @uri
    end

    assert_equal 'Content-Length (5) does not match response body length (4)' \
      ' (Mechanize::ResponseReadError)', e.message
  end

Mechanize::ResponseReadError - Content-Length does not match response body length

2 Answers2