0

I am trying to stream larger payloads with node pg-query-stream.

I am piping the JSONStream directly into my HTTP Response as process.stdout in this example.

const pg = require('pg')
var pool = new pg.Pool()
const QueryStream = require('pg-query-stream')
const JSONStream = require('JSONStream')

//pipe 1,000,000 rows to stdout without blowing up your memory usage
pool.connect((err, client, done) => {
  if (err) throw err
  const query = new QueryStream('SELECT * FROM generate_series(0, $1) num', [1000000])
  const stream = client.query(query)
  //release the client when the stream is finished
  stream.on('end', done)
  stream.pipe(JSONStream.stringify()).pipe(process.stdout)
})

This works well, however I am still hitting the Lambda payload limit.

Why would the payload limit apply to a stream?

You can see the stream working here... https://pg-query-stream.vercel.app/api/query?template=select_n&n=1000&stream=true

Payload is about ~220k.

Increase n to 10000 and the payload is about ~2.2mb.

After n = 21000 ~4.7mb I hit the payload limit and the lambda throws a 413.

Dennis Bauszus
  • 1,624
  • 2
  • 19
  • 44
  • 2
    Because lambda does not support (HTTP) streaming at all. Your code attempting to stream will still result in one big payload blob the lambda actually attempts to return. And that will obviously fail because you hit the limit. – luk2302 Mar 10 '23 at 11:24

0 Answers0