0

the question is very simple. I am in an azure function environment using python as programming language. My goal is to create a nd-json because I have a huge result and I want it to be returned to the caller as nd-json in a stream so I can easily consume it chunk by chunk or better line by line.

The problem is that the

import azure.functions as func
func.HttpResponse

that has to be returned does only accept bytes, string or integer. There is no possibility to put a generator here.

I read on stackoverflow somebody who suggested to use a django StreamingHttpResponse. However this also does not help because in the end I have to encapslulate it into the HttpResponse object from python.

So is there really no way in azure function context to create a real nd-json ? This would be really a shame because I don't want to use orchestration or some other overly sophisticated solution when I could just stream my data.

tuxmania
  • 906
  • 2
  • 9
  • 28
  • Define “stream” – Skin May 15 '23 at 08:03
  • Stream as in nd-json style that means i want to calculate a json response object, then push it out to the response and then calculate the next. I dont want to wait until i potentially created millions of lines and store them all into memory and then push them out. The idea of nd-json is to not do that and only use memory for one object that you are putting together then dump it into the stream, flush and calculate the next one. – tuxmania May 15 '23 at 18:53

1 Answers1

0

If you do not want to use HttpStreamingResponse you can make use of the code below to get the response as stream with your Azure Functions Http Trigger Python code:-

My init.py:-

import azure.functions as func

def generate_ndjson():
    # Your code to generate the ND-JSON data
    yield '{"key1": "value1"}\n'
    yield '{"key2": "value2"}\n'
    yield '{"key3": "value3"}\n'
    # ...

def ndjson_response(req: func.HttpRequest) -> func.HttpResponse:
    ndjson_data = list(generate_ndjson())  # Collect the generated data into a list
    response = func.HttpResponse('\n'.join(ndjson_data), status_code=200, mimetype='application/x-ndjson')
    response.headers['Content-Length'] = str(len(response.get_body()))

    return response

def main(req: func.HttpRequest) -> func.HttpResponse:
    return ndjson_response(req)

Output:-

enter image description here

enter image description here

SiddheshDesai
  • 3,668
  • 1
  • 2
  • 11
  • the '\n'.join(ndjson_data) part is making it blocking call or ? my goal is to push the value out as soon as it is yielded. In your repsonse first the generator is creating all the data and afterwards the ndjson_response() function is terminating bringing everything at once. I don't see how '\n'.join can be non blocking – tuxmania May 15 '23 at 18:52