httpx.RemoteProtocolError while processign 1Gb+ file

Question

My code downloads huge file with httpx and process it's chunks on the fly

async with httpx.AsyncClient() as client:
    async with client.stream("GET", self.url, follow_redirects=True, timeout=60) as stream:
        async for chunk in stream.aiter_text():
            parser.feed(chunk)
            await self._process_element(parser)

When I run it on my notebook it works good but in the the kuber cluster on dedicated pod I have got error: httpx.RemoteProtocolError: peer closed connection without sending complete message body after about 300K _process_element()'s.

Doc for h11 says, that "maximum number of bytes we’re willing to buffer of an incomplete event. In practice this mostly sets a limit on the maximum size of the request/response line + headers. If this is exceeded, then next_event() will raise RemoteProtocolError." Does it means my code working too slow and cannot manage incoming stream? And the 2nd: can I increase buffer for incoming stream? There is no interface for this in HTTPX as far as I know.

Any advises welcome. Thank you.

score 0 · Answer 1 · answered Jun 20 '23 at 10:56

all of httpx network functionality is handled by httpcore, a separate project. Anyway, there is a very dirty way to do it, which I do not recommend, but you can use it for debugging purposes because httpcore does not provide any API to do it the normal way.

import httpcore

httpcore._async.http11.AsyncHTTP11Connection.MAX_INCOMPLETE_EVENT_SIZE = 10**10
httpcore._sync.http11.HTTP11Connection.MAX_INCOMPLETE_EVENT_SIZE = 10**10

pool = httpcore.ConnectionPool()
pool.request("GET", "https://www.youtube.com")
print(pool.connections[0]._connection._h11_state._max_incomplete_event_size) 
# 10000000000

with httpx

import httpcore
import httpx

httpcore._async.http11.AsyncHTTP11Connection.MAX_INCOMPLETE_EVENT_SIZE = 10**10
httpcore._sync.http11.HTTP11Connection.MAX_INCOMPLETE_EVENT_SIZE = 10**10

pool = httpx.Client()
pool.request("GET", "https://www.youtube.com")
print(pool)

If that helps, you can open an issue in the httpcore issue tracker, and we will provide a user-friendly interface for changing that behavior.

httpx.RemoteProtocolError while processign 1Gb+ file

1 Answers1