Iterable object and Django StreamingHttpResponse

Question

I want to connect to internal http services with django and I need to buffer the output the http response of those services because some contents are very large.

I am using python 3.6, django 2.0, http.client and the following code:

class HTTPStreamIterAndClose():
    def __init__(self, conn, res, buffsize):
        self.conn = conn
        self.res = res
        self.buffsize = buffsize
        self.length = 1

        bytes_length = int(res.getheader('Content-Length'))

        if buffsize < bytes_length:
            self.length = math.ceil(bytes_length/buffsize)

    def __iter__(self):
        return self

    def __next__(self):
        buff = self.res.read(self.buffsize)

        if buff is b'':
            self.res.close()
            self.conn.close()

            raise StopIteration
        else:

            return buff

    def __len__(self):
        return self.length


def passthru_http_service(request, server, timeout, path):
    serv = HTTPService(server, timeout)
    res = serv.request(path)

    response = StreamingHttpResponse(
        HTTPStreamIterAndClose(serv.connection, res, 200),
        content_type='application/json'
    )
    response['Content-Length'] = res.getheader('Content-Length')

    return response

And the reponse is empty, I test the iterator with:

b''.join(HTTPStreamIterAndClose(serv.connection, res, 200)

And everything works fine, I don't know why is not working.

What do you mean "not working"? Do you get http 500? 200 with a blank output? Provide `curl -v http://localhost:8000/x/y/` (or Postman or ...) output in your question please. — Arman Ordookhani, Jan 03 '18 at 15:32
Use an external connection as client, you will be see really "what happened" ! According to your question, it is impossible to connect to a loopback address this way (define another loopback for the stream job). — dsgdfg, Jan 03 '18 at 19:58
That may seem a bit of a long-shot @FelipeBuccioni but can you try and change the `content_type` to `text/event-stream` (or even remove it all together)? — John Moutafis, Jan 03 '18 at 20:55
What WSGI server are you using? Is there a nginx-like proxy in front of it? — Oleg Kuralenko, Jan 07 '18 at 20:10

score 4 · Answer 1 · edited Oct 07 '21 at 08:58

https://andrewbrookins.com/django/how-does-djangos-streaminghttpresponse-work-exactly/

First, some conditions must be true:

The client must be speaking HTTP/1.1 or newer

The request method wasn’t a HEAD

The response does not include a Content-Length header

The response status wasn’t 204 or 304

If these conditions are true, then Gunicorn will add a Transfer-Encoding: chunked header to the response, signaling to the client that the response will stream in chunks.

In fact, Gunicorn will respond with Transfer-Encoding: chunked even if you used an HttpResponse, if those conditions are true!

To really stream a response, that is, to send it to the client in pieces, the conditions must be true, and your response needs to be an iterable with multiple items.

Basically, you need to decide: streaming or Content-Length.

If you want resumable downloads use Range.

Does this apply to other WSGI servers as well? – dahrens Jan 06 '18 at 20:14 — dahrens, Jan 06 '18 at 20:14
IDK, I didn't experiment. – Krzysztof Szularz Jan 09 '18 at 13:15 — Krzysztof Szularz, Jan 09 '18 at 13:15

score 0 · Answer 2 · answered Jan 15 '18 at 21:35

At the end, the problem was in the fact that the http request disconnects after a some ms, thats why when I iterate inmediatly after the connection and before the creation of the objects works find and not when I use after the response, but when I start the connection when start to iterate, all works perfectly =/.

Cheers.

Iterable object and Django StreamingHttpResponse

2 Answers2