How can i properly send a stream from forked child process in Node.js?

Question

I tried something like this in the forked child process:

WriterPlugin.js (child process)

  function sendCursor(){
                        try {
                let cursor = await getQueryCursor(reqBody, db);
                cursor.on('data', (data) => {
                    process.send(data);
                })
            } catch (error) {
                process.send({
                    message: error.toString(),
                    status: 400
                })
            }
        }

controller.js (parent process)


const childProcess = fork(fileToExec);
    childProcess.send(objectToProcess);
    childProcess.on('message', (data) => {
            reply.send(data);
    })

This one printed just last data of cursor and i faced with a fastifyError:

"code":"FST_ERR_REP_ALREADY_SENT","statusCode":500},"msg":"Reply already sent"}

How can i properly handle the cursor.stream() from a forked child process using fastify?

@ApoorvaChikara I have tried in various ways but or i faced with "dest.on is not a function" or i was able to print just last record. Any idea? — Gus Evan, May 18 '21 at 13:19
You need to share little more code and error stack to help you out. — Apoorva Chikara, May 18 '21 at 13:20
as @ApoorvaChikara suggested, i added some extra code and the stacktrace. I Hope this help — Gus Evan, May 18 '21 at 13:42
It is little difficult where the problem might be in the code , but error says that response already being sent, so check if you are sending response twice somewhere. — Apoorva Chikara, May 18 '21 at 13:54
Yes i know, because i'm sending during stream for this i'm looking for alternative solution. — Gus Evan, May 18 '21 at 14:05

score 1 · Accepted Answer · answered May 18 '21 at 15:59

You need to push the data into a stream to accomplish this task:

const { fork } = require('child_process')
const { Readable } = require('stream')

const fastify = require('fastify')()

fastify.get('/', (request, reply) => {
  const stream = new Readable({
    read (size) {}
  })

  const childProcess = fork('./external-file.js')
  childProcess.on('message', (data) => {
    stream.push(JSON.stringify(data)) // it must be a string
  })
  childProcess.on('close', (data) => {
    stream.push(null)
  })

  reply.send(stream)
})

fastify.listen(8080)

Where external-file.js is:

let iteration = 0
const timer = setInterval(() => {
  const data = { iteration: iteration++ }
  process.send(data)

  if (iteration === 3) {
    clearInterval(timer)
  }
}, 1000)

This answer was a good starting point for properly handling the stream from the child process. I encountered a problem sending the promise from the child process because i was sending data through process.send() in cursor.on event, but the parent could not know when the son had finished, so reply.send started before the resolution of the stream and process.on. In externalFile.js i implemented cursor.on('close) to send process .exit and works fine. — Gus Evan, May 19 '21 at 09:09

How can i properly send a stream from forked child process in Node.js?

1 Answers1