Stdout buffer issue using node child_process

Question

I'm trying to execute curl using node child_process to get a JSON file (about 220Ko) from a shared folder in a local network. But it actually returns a buffer problem that I can't get throught. Here is my code:

var exec = require('child_process').exec;

var execute = function(command, callback){
    exec(command, function(error, stdout, stderr){ callback(error, stdout); });
};

execute("curl http://" + ip + "/file.json", function(err, json, outerr) {
    if(err) throw err;
    console.log(json);
})

And here is the error I get:

if(err) throw err;
          ^
Error: stdout maxBuffer exceeded.
    at Socket.<anonymous> (child_process.js:678:13)
    at Socket.EventEmitter.emit (events.js:95:17)
    at Socket.<anonymous> (_stream_readable.js:746:14)
    at Socket.EventEmitter.emit (events.js:92:17)
    at emitReadable_ (_stream_readable.js:408:10)
    at emitReadable (_stream_readable.js:404:5)
    at readableAddChunk (_stream_readable.js:165:9)
    at Socket.Readable.push (_stream_readable.js:127:10)
    at Pipe.onread (net.js:526:21)

score 203 · Accepted Answer · 2014-05-02T13:54:23.863

203

You need to use and set the maxBuffer option when using child_process.exec. From the documentation:

maxBuffer specifies the largest amount of data allowed on stdout or stderr - if this value is exceeded then the child process is killed.

The documentation also states that the default value of maxBuffer is 200KB.

As an example, the maximum buffer size is increased to 500KB in the following code:

var execute = function(command, callback){
    exec(command, {maxBuffer: 1024 * 500}, function(error, stdout, stderr){ callback(error, stdout); });
};

Additionally, you may want to read about http.get to see if it is capable of achieving what you are trying to do.

edited May 02 '14 at 13:54

answered May 02 '14 at 13:43

This solved my problem, thanks! The shared folder is actually under webdav protocol that requires a digest authentication, that's why i'm using curl which handles it very easily with `curl --digest http://login:password@" + ip + "/webdav/file.json` – Yonnaled May 02 '14 at 14:08
2

This default is ridiculously small. This is the second time I was bitten by this in a hard-to-find way. – jlh Nov 22 '18 at 14:42
6

The default is now 1MB @jlh, https://nodejs.org/api/child_process.html#child_process_child_process_exec_command_options_callback – Carlos Dec 17 '19 at 12:53
3

A little trick. If you set `{maxBuffer: undefined}` then there will no limit to the buffer size. Internally `child_process` allocates small buffer chunks as the command produces output, so it never preallocate a `maxBuffer`-sized buffer, but rather grows it as needed. `maxBuffer` is meant only as a fail-safe to stop a process that's outputting too much. `undefined` happens to slips through the validation checks and also circumvents the fail safe. It's not documented anywhere and may stop working at any time. Use with care. – Allon Guralnek Jul 11 '22 at 07:26
to be clear: `maxBuffer` is a copy. OS' allocate an internal buffer for `stdout` per their own rules. If `proc` would overfill that buffer, it's suspended. Assuming the emit requirements by nodeJS are met (which I believe is a simple newline character anywhere in the buffer?), `subprocess.stdout.on` will fetch a chunk of the buffer, fire it at the handler, and `free` that chunk, letting the OS resume the producing subprocess. I believe the C API's for stdout require that it be simply a cbuf[] with some indicies, universally exposed as a circular array. Is this all correct? – Groostav Jul 24 '23 at 22:37

score 67 · Answer 2 · answered Nov 18 '14 at 13:22

67

I had a similar issue and I fixed it moving from exec to spawn:

var child = process.spawn('<process>', [<arg1>, <arg2>]);

child.stdout.on('data', function (data) {
  console.log('stdout: ' + data);
});

child.stderr.on('data', function (data) {
  console.log('stderr: ' + data);
});

child.on('close', function (code) {
    console.log('child process exited with code ' + code);
});

answered Nov 18 '14 at 13:22

lsampaio

908
7
7

https://nodejs.org/api/child_process.html#child_process_child_process_spawn_command_args_options – cs01 Jun 19 '16 at 01:59
3

This answer is not necessarily the most appropriate one. I think console output in the question could be just an example. Hardly someone would fetch a 200KB file to throw it to console. However, if `process.exec` is used in things like CLI tools, then yes, switching to `spawn` should be the way to go. – Pavel Gatilov Jun 27 '18 at 14:59
2

wow... spawn is cool. It's not even using call backs or promises... just events. That could be really useful for streaming stdout to the console. @Pavel Gatilov, that is exactly what we're doing. FFMpeg likes to show progress every second... which takes a toll on the buffer – Ray Foss Jun 07 '19 at 21:22
1

This event listener pattern can be used in exactly the same way with `exec` instead of `spawn` (just don't use a callback in `exec`). I prefer using `exec` since it does not require splitting up arguments into a separate array. – derpedy-doo Sep 14 '21 at 02:18

score 7 · Answer 3 · answered Dec 11 '20 at 09:16

Adding some explanation to the answers.

The exec command buffers the data before sending it to the parent process. It is generally suitable for the commands producing the smaller output. The above error occurred because the output generated by the execution of the command was larger than the max buffer size. One way to solve the above issue is to specify the buffer size as answered by Tim Cooper.

var execute = function(command, callback){
exec(command, {maxBuffer: 1024 * 500}, function(error, stdout, stderr){ 
 callback(error, stdout); });
};

Another solution is to use the spawn method which is generally faster compared to the exec and it does not buffer the data before sending. It sends the data as a stream hence the problem of buffer size never occurs. The code snippet used by Isampaio.

var child = process.spawn('<process>', [<arg1>, <arg2>]);
child.stdout.on('data', function (data) {
 console.log('stdout: ' + data);
});
child.stderr.on('data', function (data) {
 console.log('stderr: ' + data);
});
child.on('close', function (code) {
 console.log('child process exited with code ' + code);
});

GBra 4.669 · Answer 4 · 2022-07-15T11:19:40.443

0

Fastest solution ever:

If this line outputs a buffer:

childProc.stdout.on('data', (data) => console.log(data)); // <Buffer 1b 5b 34 6d>

You just change it to:

childProc.stdout.on('data', (data) => console.log(`${data}`)); // bla bla bla

JS console will encode your buffer.

Or we can simply use: console.log(data.toString()).

edited Jul 15 '22 at 11:19

answered Jul 15 '22 at 11:12

GBra 4.669

1,512
3
11
23

Stdout buffer issue using node child_process

4 Answers4

Linked