stdout always cuts off the stream at character 8192

Question

I am using spawn module and executing curl command and getting JSON response from server. This is my code -

var spawn = require('child_process').spawn;
log.debug("Executing curl command for getting all artifacts");

var artifacts_data = spawn('bash module_features/init_get_all_artifacts.sh', [
    host_url, access_token, project_id
], {
        shell: true
   });
artifacts_data.stdout.setEncoding('utf8');
var stdout = "";
artifacts_data.stdout.on('data', (data) => {
    stdout += data.toString();
    try {
        var artifact_array = JSON.parse(stdout);
        log.debug(artifact_array);
        if (artifact_array != null || artifact_array != undefined){
            res.send(200, { status: 'success', message: artifact_array });
        }else{
            log.debug('Artifact list not found.');
            res.send(400, { status: 'error', message: 'In else :: somthing went wrong' });
        }
    } catch (error) {
        log.debug("There was some error in parsing artifacts JSON");
        res.send(400, { status: 'error', message: 'somthing went wrong' });
    }

});

But, I got half string (8192 characters) and which is incorrect JSON. Please help me to increase character limit of stdout or any alternative solution.

Any help is appreciated.

score 0 · Accepted Answer · answered Mar 08 '18 at 07:20

And I fixed the issue.

var data = '';
artifacts_data.stdout.on('data',(data) => {
    stdout += data.toString();
});

artifacts_data.stdout.on('end',(data) => {
    try {
         var artifact_array = JSON.parse(stdout);
         log.debug(artifact_array);
         if (artifact_array != null || artifact_array != undefined){
            res.send(200, { status: 'success', message: artifact_array });
         }else{
              log.debug('Artifact list not found.');
              res.send(400, { status: 'error', message: 'In else :: somthing went wrong' });
         }
    } catch (error) {
          log.debug("There was some error in parsing artifacts JSON");
          res.send(400, { status: 'error', message: 'somthing went wrong' });
    }

});

If we are dealing with IO in a web application we have to stick with the async methods.

score 0 · Answer 2 · answered Nov 21 '19 at 15:32

The problem with end event suggested by iSmita is that it trigger only when the process exit. So you lose in reactivity, especially if you expect more than one answer.

If the message end with \n you can do something more generic like this :

let stdoutBuffer = ""
// Stdout messages catching
process.stdout.on('data', (data) => {
  data = data.toString() // Convert chunk to string

  // Add too buffer but no process util the end
  stdoutBuffer += data

  // Split if several lines at once
  const answers = stdoutBuffer.split('\n')

  // Send the last part to the buffer.
  // If the end was '\n' so the buffer will be set back with an empty string.
  // If their is no '\n' so the buffer won't be consumed and answers will be empty.
  stdoutBuffer = answers.pop()

  answers.forEach((a) => {
    // do some stuff with the answer
  })
})

stdout always cuts off the stream at character 8192

2 Answers2