Pipelining POST requests with python-requests

Question

Assuming that I can verify that a bunch of POST requests are in fact logically independent, how can I set up HTTP pipelining using python-requests and force it to allow POST requests in the pipeline?

Does someone have a working example?

P.S. for extra points, how to handle errors for outstanding requests if pipeline suddenly breaks?

P.P.S. grequests is not an option in this case.

Can you elaborate what you mean by a *pipeline*? Are you talking about [HTTP pipelining](http://en.wikipedia.org/wiki/HTTP_pipelining) here? *Why* is `grequests` not an option? — Martijn Pieters, Mar 16 '15 at 12:53
Note the `requests` does [not support HTTP pipelining](https://github.com/kennethreitz/requests/issues/1394); using `grequests` would only ever be an approximation. — Martijn Pieters, Mar 16 '15 at 12:57
@MartijnPieters oh, my google-fu hath failt me... Please convert that into an answer for acceptance. — Dima Tisnek, Mar 16 '15 at 12:59

score 3 · Answer 1 · edited Nov 25 '21 at 00:42

3

Pipelining requests can be done with the builtin httplib, but only by accessing the connection and response objects below their public interface. This snippet demonstrates.

Edit: updated version for Python3: https://github.com/urllib3/urllib3/issues/52#issuecomment-109756116

edited Nov 25 '21 at 00:42

Dima Tisnek

11,241
4
68
120

answered May 29 '15 at 00:37

A. Coady

54,452
8
34
40

score 2 · Accepted Answer · answered Mar 16 '15 at 13:03

The requests library does not support HTTP pipelining.

You can approximate pipelining by using grequests which makes it easier to run many requests in parallel, but each parallel request would still use a new TCP connection.

(requests does pool connections, keeping the TCP connection open if the remote server permits this, but that only helps for sequential connections, and request and response still have to alternate).

Pipelining POST requests with python-requests

2 Answers2

Linked