0

I have a result set as a stream of Arrow record batches,i have used reader.read_chunk() to get the batchs ,i have pushed the batches to batch array and convert it to bytearray as response .for that here is the code

def getBatchStreambytes(_):
   reader = client.do_get(flight_info.endpoints[0].ticket, options)
            print('[INFO] Reading query results from Dremio Server ')
            batches = [] 
            while True:
                     try:
                            batch, metadata = reader.read_chunk()
                            print(batch.num_rows)
                            batches.append(batch)

                except Exception as exception:
                        break

        data = pa.Table.from_batches(batches)
        sink = pa.BufferOutputStream()
        writer = pa.RecordBatchStreamWriter(sink, data.schema)
        writer.write_table(data)
        writer.close()
        #print(reader.read_pandas())

        return  sink.getvalue().to_pybytes()

as API response taking more time for each batches ,how its possible to send batch as response so that api will respond with iterated batch ,how i can send chunks of batch. Here us screenshots of List of Batches i recived from flight server enter image description here

vikashsingh9
  • 41
  • 7
  • It's not clear to me what you are asking. You are currently reading data one batch at a time. You are asking how to send the batch as a response. However, I do not know you mean. Is this some kind of HTTP server and you want to send a response? Are you writing a flight server? Is `getBatchStreambytes` a standard method of some library? I'm afraid it is not very clear to me what you are doing. – Pace May 11 '21 at 19:23
  • i want to send response as chunk in api response as if the query has over 50 million rows or so then in that case api tooks 2 -3 min resonse time.how i can send chunk of batch as response so atlest even 10 record in batch i can get 10 record as response .I am not writing any flight server ,i am using dremio which is already flight enabled server – vikashsingh9 May 12 '21 at 13:15

0 Answers0