Node Kue and Child Process - get error from spawned process

Question

I try to spawn a child process performing cpu intensive calculations through a job queue with Kue. My code at the moment looks like this:

consumer.js

var kue = require('kue');
var util  = require('util');
var spawn = require('child_process').spawn;

var jobs = kue.createQueue();

jobs.process('calc', 2, function(job, done){
  var work = spawn('Rscript', ['opti2.R', job.data.file]);

  work.stderr.on('data', function (data) {
    job.log('stderr: ' + data);
  });

  work.stdout.on('data', function (data) {
    job.log('stdout: ' + data);
  });

  work.on('exit', function (code, signal) {
    console.log('child process exited with code ' + code + ' with singal ' + signal);
    if(code != 0){
      done(****How to get the stderr of the child process as an error here***);
    } else {
      done(Error());
    }
  });
});

The code somewhat do what i would like it to do, but is there a better way to report the job as failed (to Kue) and get the stderr from the spawned process?

Do you want child's `stdio` to be piped to parent's `stdio`, or do you want it to be sent to `Kue`? — Leonid Beschastny, Aug 17 '14 at 00:33
@LeonidBeschastny: Not sure (very new to node/programming), but would like to have the error from the spawn process as the error/failed message in Kue. What do you recommend? — James Bund, Aug 17 '14 at 00:48

Leonid Beschastny · Accepted Answer · 2014-08-17T11:34:15.893

You can use job.log method to send data directly to Kue.

I would also recommend you to switch from .spawn to .exec, because it returns stdout and stderr as strings in its final callback along with a good error, which suits your needs well:

var exec = require('child_process').exec;

jobs.process('calc', 2, function(job, done){
  exec('Rscript opti2.R ' + job.data.file, function (error, stdout, stderr) {
    if (stdout.length > 0) job.log('stdout: ' + stdout);
    if (stderr.length > 0) job.log('stderr: ' + stderr);
    done(error);
  });
});

Though solution should work with .spawn as well: simply replace each console.log call in your code with job.log.

Though, you may want to bufferize your stderr in order to send it to Kue in one chunk:

jobs.process('calc', 2, function(job, done){
  var work = spawn('Rscript', ['opti2.R', job.data.file]);
  var stderr = '';

  work.stderr.on('data', function (data) {
    stderr += data;
  });

  work.stdout.on('data', function (data) {
    job.log(data); // sending arriving `stdout` chunks as normal log events
  });

  work.on('close', function (code, signal) {
    console.log('child process exited with code ' + code + ' with singal ' + signal);
    if(code != 0){
      done(stderr); // sending all collected stderr as an explanation
    } else {
      done();
    }
  });
});

I would also recommend using close event instead of exit, because it waits for child's stdio streams.

For more information see Event: 'exit' docs:

This event is emitted after the child process ends.

Note that the child process stdio streams might still be open.

and Event: 'close' docs:

This event is emitted when the stdio streams of a child process have all terminated.

The calculations runs for 30-90 mins and outputs (stdout) along the way, which i would like to return to the webpage, therefor i use spawn. Is there a way to carry the error from the stderr event to the done() of Kue? — James Bund, Aug 17 '14 at 10:26
@JamesBund as I said, this solution will work with `.spawn` as well, just use `job.log` to send data to `Kue` instead of logging it to the console. — Leonid Beschastny, Aug 17 '14 at 10:29
Not sure if misunderstand something, but with job.log I'm still not able to use the stderr from the child as a argument to the done() in the exit event of the child? Have updated the code.. — James Bund, Aug 17 '14 at 11:06
@JamesBund The only way to get the full `stderr` in `.on 'exit'` callback is to bufferize it manually. Though, your edited code should allow you to see child's `stderr` in `Kue` without passing it to `done()` callback. — Leonid Beschastny, Aug 17 '14 at 11:12
But if i don't pass it to the done(), it want be part of 'explanation'/'error trace back' of the failed event, just a normal log event..? How can i bufferize it? — James Bund, Aug 17 '14 at 11:18

Node Kue and Child Process - get error from spawned process

1 Answers1