Django StreamingHttpResponse returns only a single yield result from generator to stream

Question

I am trying to stream several large json files to the front end at a time (I need to stream all of them one by one, and can't split it into several requests):

def stream_response_generator(response):
    yield json.dumps({ 'searchedVariants': response['searchedVariants']})
    yield json.dumps({ 'genesById': response['genesById']})
    yield json.dumps({ 'locusListsByGuid': response['locusListsByGuid']})
    yield json.dumps({ 'genesQcData': response['genesQcData']})

@login_required(login_url=API_LOGIN_REQUIRED_URL)
@csrf_exempt
@log_request(name='query_variants_handler')
def query_variants_handler(request, search_hash):
    ...
    return StreamingHttpResponse(stream_response_generator(response), content_type='application/octet-stream')

Only the very first searchedVariants response is streaming to the front end but other 3 are omitted. I verified it printing out the response on the frontend:

fetch(url, {
    method: 'POST',
    credentials: 'include',
    body: JSON.stringify(search),
    signal: signal,
}).then((response) => {
  const reader = response.body.getReader()
  let decoder = new TextDecoder('utf8');
  let read
  reader.read().then(read = (result) => {
    if (result.done) return
    let chunk = decoder.decode(result.value);
    console.log(chunk)

    reader.read().then(read)
  })  
})

In some tutorials, e.g.

https://andrewbrookins.com/django/how-does-djangos-streaminghttpresponse-work-exactly/

Its written that we should pass a function generator instead of calling a function like I do, however, when I do that:

def stream_response_generator(response):
    def generator():
        yield json.dumps({ 'searchedVariants': response['searchedVariants']})
        yield json.dumps({ 'genesById': response['genesById']})
        yield json.dumps({ 'locusListsByGuid': response['locusListsByGuid']})
        yield json.dumps({ 'genesQcData': response['genesQcData']})

return generator

@login_required(login_url=API_LOGIN_REQUIRED_URL)
@csrf_exempt
@log_request(name='query_variants_handler')
def query_variants_handler(request, search_hash):
    ...
    return StreamingHttpResponse(stream_response_generator(response), content_type='application/octet-stream')

I am getting an error:

2022-10-21 03:34:28,165 ERROR:  Traceback (most recent call last):
  File "/site-packages/django/core/handlers/base.py", line 179, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/python3.6/site-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/gris/decorators.py", line 26, in wrapper
    return f(request, *args, **kwargs)
  File "/variant_search_api.py", line 141, in query_variants_handler
    return StreamingHttpResponse(stream_response_generator(response), content_type='application/octet-stream')
  File "/python3.6/site-packages/django/http/response.py", line 367, in __init__
    self.streaming_content = streaming_content
  File "/python3.6/site-packages/django/http/response.py", line 382, in streaming_content
    self._set_streaming_content(value)
  File "/python3.6/site-packages/django/http/response.py", line 386, in _set_streaming_content
    self._iterator = iter(value)
TypeError: 'function' object is not iterable

I suspect it is somehow related to @csrf_exempt decorator but not sure. So, why does StreamingHttpResponse not stream other values? How to fix it?

Django StreamingHttpResponse returns only a single yield result from generator to stream

0 Answers0