1

UPDATE: The kind folks developing Requests.py are on the case.

We're interacting (testing) with a HTTP service that responds with a 411 (Length Required) status code when no Content-Length request header has been supplied in the presence of a request body; This is to be expected as per RFC 2616.

The problem we've encountered centres around the Requests Python Module (curl works as expected). When sending an illformed request (lacking Content-Length in the presence of a request body) the service responds 411 and the connection is closed by the remote endpoint (Connection: close). This in turn throws an exception from the Requests Session.send method and we're unable to retrieve the HTTP response status code for validation.

Is it possible to stop this situation (exception) and retrieve the HTTP response status code without altering the HTTP service?

Code

from requests import Request, Session

endpoint = "http://api.corvusoft.co.uk/resources"
request_body = '{ "data": {"core-temperature": "1,430", "core-temperature-units": "celsius", ... } }'

request = Request('POST', endpoint, data=request_body, headers={"Connection": "close"})
prepared_request = request.prepare()
del prepared_request.headers['Content-Length']

session = Session()
response = session.send(prepared_request)

print response.status_code

Stack Trace

Scenario: HTTP PUT without Content-Length                                                                                               
    Given I have initialised the core sensors
    And I perform a HTTP "POST" request to "/readings" with headers "Content-Type: application/vnd.api+json; charset=utf-8, Accept: application/vnd.api+json":
      | """                                                   |
      | " { "data": {                                       " |
      | "     "id": "c0f22109-d258-4458-a3f5-0d16b2f55487", " |
      | "     "value": "acceptance-test"                    " |
      | "   }                                               " |
      | " }                                                 " |
      | """                                                   |
    When I perform a HTTP "PUT" request to "/queues/c0f22109-d258-4458-a3f5-0d16b2f55487" without header "Content-Length":
      | """                                                   |
      | " { "data": {                                       " |
      | "     "ttl": 30,                                    " |
      | "     "id": "c0f22109-d258-4458-a3f5-0d16b2f55487"  " |
      | "   }                                               " |
      | " }                                                 " |
      | """                                                   |
    Traceback (most recent call last):
      File "/Library/Python/2.7/site-packages/lettuce/core.py", line 144, in __call__
        ret = self.function(self.step, *args, **kw)
      File "/Users/Corvusoft/Development/test/acceptance/features/step_definitions/request.py", line 30, in i_perform_a_http_method_request_to_path_without_header_and_body
        i_perform_a_http_method_request_to_path_with_headers_and_body( step, method, path, "Accept: application/vnd.api+json, Content-Type: application/vnd.api+json; charset=utf-8", True )
      File "/Users/Corvusoft/Development/test/acceptance/features/step_definitions/request.py", line 91, in i_perform_a_http_method_request_to_path_with_headers_and_body
        response = session.send( request )
      File "/Library/Python/2.7/site-packages/requests/sessions.py", line 573, in send
        r = adapter.send(request, **kwargs)
      File "/Library/Python/2.7/site-packages/requests/adapters.py", line 415, in send
        raise ConnectionError(err, request=request)
    ConnectionError: [Errno 32] Broken pipe

Request

POST /resources HTTP/1.1
Host: api.corvusoft.co.uk
Connection: close
Content-Type: application/vnd.api+json; charset=utf-8
Accept: application/vnd.api+json
Accept-Charset: utf-8

{ "data": {"core-temperature": "1,430", "core-temperature-units": "celsius", ... } }

Response

HTTP/1.1 411 Length Required
Connection: close
Content-Type: application/vnd.api+json; charset=utf-8
Content-Length: 304

{"errors":[{"status":"411","title":"Content Length Required", "description": "The server refuses to accept the request without a defined Content-Length. The client MAY repeat the request if it adds a valid Content-Length header field containing the length of the message-body in the request message."}]}

Environment

Python: 2.7.5

Requests: 2.5.3

Ben Crowhurst
  • 8,204
  • 6
  • 48
  • 78
  • Could you add the traceback you get for this error too? – Martijn Pieters Mar 13 '15 at 10:58
  • @MartijnPieters Great idea :D, please hold... – Ben Crowhurst Mar 13 '15 at 11:02
  • If I substitute your endpoint for `http://httpbin.org/status/411` everything works just fine. That connection *is* closed (`Connection: close` is set on the response). As such I'm having a hard time reproducing your problem. Was that traceback still forthcoming? What version of `requests` are you using? (`import requests; print requests.__version__` will give you the current version). – Martijn Pieters Mar 13 '15 at 12:16
  • @MartijnPieters Traceback is on the way. I'm required to send it off to the client for sanitising before public display; don't ask! Will post asap. thx – Ben Crowhurst Mar 13 '15 at 12:38
  • @MartijnPieters The Connection header from httpbin.org is keep-alive. The remote endpoint (httpbin.org) is not closing the connection. Therefore Requests is not displaying the same behaviour. – Ben Crowhurst Mar 13 '15 at 12:40
  • Ick, so it is; the example on http://httpbin.org/ implied it'd be closed. My mistake for not double-checking. You can still force it with a `Connection: close` request header, but `requests`/`urllib3` probably alters behaviour then. – Martijn Pieters Mar 13 '15 at 12:42
  • @MartijnPieters adding Connection: close to the request will cause httpbin.org to respond with the correct header. I'll update my example to includes this header. – Ben Crowhurst Mar 13 '15 at 12:43
  • I see you are getting the `requests` / `urllib3` core devs on the case now; I see that the remote server closes the connection as soon as the request headers are sent without waiting for the body, probably a valid way to respond. – Martijn Pieters Mar 14 '15 at 13:38
  • @MartijnPieters Correct. The Requests core team have taken up the case and hopefully will see a fix shortly. – Ben Crowhurst Mar 16 '15 at 15:00
  • If this is being discussed in Requests forum or such can you paste link here? – shreyas Jul 14 '15 at 06:17
  • Please see my latest update. – Ben Crowhurst Jul 14 '15 at 09:41
  • Did this get solved ? – Gupta Oct 08 '21 at 20:27
  • I've updated the question to fix the broken link (https://github.com/psf/requests/issues/2490); no progress appears to have been made. – Ben Crowhurst Oct 11 '21 at 08:42

0 Answers0