3

It seems for big enough stdout output node does not fire data events:

Example bash script:

#!/usr/bin/env sh

for i in {1..100000}; do
    echo $i
done

Running the above script from bash should prints all numbers from 1 to 100k one number per line

However when proxying it via node's exec:

#!/usr/bin/env node --harmony --harmony_destructuring --harmony_default_parameters

const {exec} = require('child_process');

let stdout = '';

const child = exec('./tmp.sh');

child.stdout.on('data', data => {
    stdout += data;
});

child.stdout.on('close', code => {
    if (code) process.exit(code);
    console.log(stdout.split('\n').slice(0, 3).join('\n'));
    console.log('...');
    console.log(stdout.split('\n').slice(-3).join('\n'));
});

For the above script, I'd expect to get:

1
2
3
...
99998
99999
100000

However when run It returns:

1
2
3
...
35984
35985

About a third of the output is truncated. Logging inside the data handler shows that only 1 data event is fired with 1 chunk

I've tried listening to the child.on('exit') event instead with the same result (also stdout.on('exit'))

It works the same when replacing exec with spawn

node --version # v5.4.1
osx 10.10
Stefan
  • 3,962
  • 4
  • 34
  • 39

1 Answers1

4

child_process.exec() buffers the standard output into a fixed size buffer. You are likely filling the buffer and thus you don't get the rest of the output.

There is a maxBuffer option you can pass to .exec() to make the buffer larger or you can use .spawn() and receive an unlimited stream of output.

See Stdout buffer issue using node child_process for related info.

Also worth reading: Difference between spawn and exec of Node.js child_process and Stdout of Node.js child_process exec is cut short.

Community
  • 1
  • 1
jfriend00
  • 683,504
  • 96
  • 985
  • 979
  • i did mention in my post that I tried replacing `spawn` with `exec` and got the same output - for exactly the buffer reasons – Stefan May 02 '16 at 21:09
  • 1
    oh nevermind, of course after I go and post on stack overflow and try again one of the things I tried before it ends up working :| goddamit. Thanks! – Stefan May 02 '16 at 21:11
  • @tak3r - Then, please show us the code you used with `.spawn()` because it should be able to receive an unlimited stdout stream. – jfriend00 May 02 '16 at 21:11