1

Documentation states that 'handleResponseEnd' is called when the response has been completely received.

http://twistedmatrix.com/documents/11.0.0/api/twisted.web.http.HTTPClient.html#handleResponseEnd

However, debugging appears to show that is not always true. When making multiple requests to the same URI, it appears to only get called for the FIRST request. When making multiple requests to different URIs, it still only appears to be called for the FIRST request.

Is this behavior desired? Making requests to the same URI multiple times almost makes sense, since the response received is generally "304 Not Modified", so the client does not receive the content multiple times. However, I would still expect for a callback to run stating "we are done receiving the response".

Is there a different callback that would be more suitable to access the full response from?

dicato
  • 684
  • 4
  • 13

2 Answers2

0

Usually you'd instantiate a new HTTPClient for each request by calling buildProtocol() on your factory. This is what HTTPPageGetter and Agent do.

One exception to this is when using HTTP persistent connections.

What are you doing such that you need to use the same protocol instance for multiple requests?

Peter Le Bek
  • 982
  • 8
  • 16
  • You are correct in terms of calling buildProtocol, which is what I do. I realized after a day that my problem was more complicated than I original thought due to the way I am using inheritance. Thank you for trying to help! – dicato Jun 16 '11 at 22:58
  • Forgetting to call the overridden parent method from your child method sounds _less_ complicated. Never mind - glad you figured it out. – Peter Le Bek Jun 16 '11 at 23:30
  • I know, right? However, part of my original question still stands. Careful reading of the documentation and experimentation shows that handleResponseEnd can be called MULTIPLE times. So I still cannot use it as a reliable event in order to ensure I am done receiving content. If you have ideas there, I am all ears. – dicato Jun 16 '11 at 23:38
  • 1
    `handleResponseEnd()` gets called both when the connection is lost, and when the number of bytes received equals the `content-length` specified in the HTTP response header. If I were you I'd call my response handling code the first time it's called, and ignore subsequent calls. I can't see a case where `handleResponseEnd()` would be called when there's still data to be received, unless an incorrect `content-length` was set by the server. – Peter Le Bek Jun 17 '11 at 00:19
  • I wonder what the author's intent was having it called more than once? I would think the connectionLost event would take care of connection lost scenario. Do you have a link to the API Doc or source that shows this in detail? Worst case scenario would be for me to simply use a flag to keep the state of how many times handleResponseEnd() is called. – dicato Jun 17 '11 at 04:08
  • You know `handleResponseEnd()` already provides the functionality I just described? `handleResponse()` is what you want. http://twistedmatrix.com/trac/browser/tags/releases/twisted-11.0.0/twisted/web/http.py#L457 – Peter Le Bek Jun 17 '11 at 07:10
  • I think you are right, but which class implements handleResponse? HTTPClient does not, and I can't see to find it by following the inheritance chain. – dicato Jun 28 '11 at 20:31
0

I actually figured it out last night. I have a child class that inherits from HTTPClient. When overriding connectionMade, I forgot to call the parent's connectionMade. Since connectionMade acts a as default constructor for a protocol, the instance was not getting properly configured.

While I don't know every detail of this solution, it has proven reliable in testing. The anomalies that were occurring are gone.

Peter Le Bek
  • 982
  • 8
  • 16
dicato
  • 684
  • 4
  • 13