Synchronize promises created asynchronously

Question

I'm working on a little nodejs program and have some troubles to know when all async operations of the program are completed.

For now, the program do the following steps:

1/ Spawn a process with some parameters. This process will print data on its stdout.

2/ Listen to the process stdout "data" event and everytime something is printed the program calls a function (which we will call "process"), using the data.

3/ This process function will eventually insert the data into a mongo database and sending a message to a amqp server.

3/ When there is no more data, the program is idling because the connection to the amqp and mongo database are still alive, so I need to know when all the work is done to be able to close the connections.

So, I tried to use when.js to make use of promises, but I can't make it work for what I'm trying to achieve.

I made the "process" function return a promise, which will be resolved when mongodb insert and amqp message sending are done. In my program, I'm creating an array which will receive all the promise to be able to call when.all() to know when they are all resolved.

But since I'm creating promises asynchronously when the spawned process is printing data to its stdout stream, the call to when.all() is made with an empty array which seem to be resolved immediately.

Here is a code sample which illustrate what I'm achieving:

var when = require('when')
  , _    = require('lodash')
  , cp   = require('child_process');

var promises = [];

function process(data) {
    var deferred = when.defer();

    setTimeout(function () {
        deferred.resolve(true);
    }, 3000); // Let's say we need 3 seconds to process and save data

    return deferred.promise;
}

var ls = cp.spawn('ls', ['-la', '/usr/bin']);
ls.stdout.on('data', function (data) {
    console.log('Received data, creating a promise to notify when this data is processed.');
    promises.push(process(data));
});

when.all(promises).then(function (values) {
    console.log('All promises are now resolved', values);
});

As you might have guessed, the output of this program is:

All promises are now resolved []
Received data, creating a promise to notify when this data is processed.
Received data, creating a promise to notify when this data is processed.
Received data, creating a promise to notify when this data is processed.
Received data, creating a promise to notify when this data is processed.
Received data, creating a promise to notify when this data is processed.
Received data, creating a promise to notify when this data is processed.
Received data, creating a promise to notify when this data is processed.
Received data, creating a promise to notify when this data is processed.
Received data, creating a promise to notify when this data is processed.

Is there any way to make this code sample print console messages in the expected order (which is having the first line printed last) ?

Thanks.

Put the `when.all()` call **inside** the `setTimeout()` handler. — Pointy, Dec 11 '13 at 14:53
I can't because otherwise it won't be compliant with the real code of the program, using a spawned process and listening to its stdout "data" event. The second setTimeout is an emulation of this event. In my real program, you have to imagine this will be called multiple times. It does not make sense to have multiple calls to when.all() does it ? Or am I missing something ? — Pierre-Yves, Dec 11 '13 at 14:58
It does not make sense to call `when.all()` before the promises exist. The promises come into existence synchronously when you *start* an asynchronous operation. I think that your "test" setup here is the real problem - your `setTimeout()` call itself is not returning a promise, but a real asynchronous operation would. — Pointy, Dec 11 '13 at 15:00
**Pointy** totally right. `Promise`, as it name said, is a _right-now_ pledge to do something in future. Conception of "promise made asyncronous" looks weird. — Tommi, Dec 11 '13 at 15:36
Thanks for your answers guys. That was helpful. I updated my code sample to reflect my real use case and as you can see, if the test setup is the issue, then the real use case is the same issue. I cannot create the promises in advance since I don't know what will be returned by the spawned process. Well, I guess I cannot know when everything is done this way. Thanks again ! — Pierre-Yves, Dec 11 '13 at 18:30

score 2 · Accepted Answer · answered Dec 11 '13 at 19:29

You need to call when only after all promises have been made. In your comment you mentioned:

I guess I cannot know when everything is done this way

Which is incorrect. You can know when everything is done using the close event:

ls.on('close',function(){
    when.all(promises).then(function (values) {
        console.log('All promises are now resolved', values);
    });
});

Synchronize promises created asynchronously

1 Answers1