Matching HTTP responses with their corresponding HTTP pipelined requests

Question

I'm trying to write a program to match HTTP requests with their corresponding responses. Seems that everything is working well for most of the scenarios (when the transfer is perfectly ordered and even when its not, by using TCP sequence numbers).

The only problem I found is for when I have pipelined requests. After that, I get several responses but I don't know which packets are the answer to a specific request and which are not. I read in another post that the responses will come sequentially and combining this property with information on the Content-Length field seems to be a solution. The problem is that Content-length is not a mandatory field, so I'm not sure if I can always rely on that.

Does anyone know how the web-browsers that support this feature (btw, not most of them do) actually do it?

Answer 1

The information about the bodies length has to be present in the headers. It's just not always in 'content-length'. In order to work it all out you will have to study the relevant RFC 2616. Most notably section 4.4 deals with the different headers

Some more relevant rules from the RFC 2616:

When pipelining:
A server MUST send its responses to those requests in the same order that the requests were received.

From 9.2
If no response body is included, the response MUST include a Content-Length field with a field-value of "0".

From 10.2.7 206 Partial Content
The response MUST include .... Either a Content-Range header field ... or a multipart/byteranges Content-Type including Content-Range fields for each part.

From 14.13 Content-Length
Applications SHOULD use this field to indicate the transfer-length of the message-body, unless this is prohibited by the rules in section 4.4.

Answer 2

Current responses are a bit old. Need a refresh.

The new HTTP 1.1 RFC is RFC 7230. And contains more precise information on parsing the messages size.

• Message Body Length
• Associating a response to a request
• Security Considerations

Detecting the size of a message is quite complex. You can have a Content-length, or Transfer-Encoding: chunked, or both, or none. And some sepcial codes like 100 Continue which may alter all this.

The first link contains 7 entries that should be checked in the right order to guess the right size.

And as stated in the last link, failing to detect the right message length may lead to HTTP Smuggling (splitting, cache poisoning) issues.

Pipelining support is the source of most smuggling issues. You should really take care of the whole RFC7230 document if you want to implement it.