Node.js/Javascript - nested promises and for loop with when.js

Question

I am currently struggling with the control flow of promise (promise newbie!).

I make a call to Redis which returns an array object. I then iterate through each of the results and call back to Redis to get a value and wish to populate these in to a final object out.

The out object is never populated, im guessing as the forEach has not completed:

(note the Redis client lib returns a when.js based promise as default)

var out = {};

REDISCLIENT.keys("apikey:*")
    .then(function (replies) {

        replies.forEach(function (key, index) {

            REDISCLIENT.get(key)
                .then(function (value) {
                    out[key] = value;
                })
                .done(console.log(out));

        });

    })
    .catch(console.log)
    .done(console.log(out));

How can I guarantee that the forEach loop is complete?

I have read many similar posts (I know this is a duplicate) however have not been able to understand why the inner done() method does not contain the fully complete out obj.

Im guessing I need to wrap the forEach itself in a promise? Appreciate any direction.

Update 1: huge thanks to @David_Aurelio. I now need to populate out with the key and values. Here is my attempt:

GLOBAL.REDISCLIENT.keys("apikey:*")
        .then(function (replies) {
            return when.all(replies.map(function (key, index) {
                return GLOBAL.REDISCLIENT.get(key)
                    .then(function (val) {
                        out[key] = val;
                        console.log(key, val);
                    });
            }));
        })
        .catch(console.log)
        .done(function (out) {console.log(out); });

The inner console.log prints the correct key/values

key1 val1
key2 val2

The final done now prints:

[ undefined, undefined ]

You are magically expecting that the outer promise knows when the inner code is done, but you've made no connection between them so the outer one doesn't wait for the inner ones to finish. Thus, none of that inner code has finished yet when you try to log the value of `out` on the last line of your code. I don't know when-js, but you'll have to create a master promise that is resolved when ALL your inner promises are done and return that from the outer `.then()` handler. This will tell the outer promise to wait for all the inner activity to be done. — jfriend00, Jul 19 '15 at 22:08
You get `[undefined, undefined]` because `out` in the .done callback is not the `out` object that was populated. Simple solution is `.done(function () {console.log(out); });`, which ensures the originally-defined `out` is logged. However, see my answer below. — Roamer-1888, Jul 20 '15 at 07:03

David Aurelio · Answer 1 · 2015-07-19T23:04:24.233

The forEach loop completes, but the Promises you create inside don’t. when.js implements the Promises/A+ spec, which guarantees asynchronous resolution of promise callbacks. That means, that the callback passed to then() is guaranteed to be invoked after the current call stack has finished to execute.

You need to return a promise from your then-callback in order to connect the inner promises to the outer promise. More specifically, you need a promise over all inner promises.

The easiest way to do that is to use when.all:

REDISCLIENT.keys("apikey:*")
  .then(function (replies) {
    return when.all(replies.map(function (key, index) {
      return REDISCLIENT.get(key);
    }));
  })
  .catch(console.log)
  .done(function (out) {console.log(out); });

In your original code, you also don't register a callback to done, but call console.log(out) immediately, before the first promise has even resolved.

@David_Aurelio - hugely helpful, thank you. Its just about there but were not appending to the out obj with the keys and values - the outer gets the keys, and each of the for loop results returns the value correctly. Please see update 1 on the question for my revised attempt. — Ben, Jul 19 '15 at 23:23

score 0 · Accepted Answer · answered Jul 20 '15 at 06:50

It's important to understand that flow control and the data conveyed by a promise chain, are determined by :

the composition of the chain, and any inner chain(s)
promise aggregators, such as when.all()
return statements

Here's how to achieve what you want with out as an inner member.

REDISCLIENT.keys("apikey:*")
.then(function (replies) {
    var out = {}: //<<<<< initiate `out` as an inner member
    return when.all(replies.map(function (key, index) { //<<<<< here's David Aurelio's when.all(replies.map(...))
        return REDISCLIENT.get(key).then(function (value) { //<<<<< `return` here causes `.map()` to build an array of promises.
            out[key] = value;
        });
    })).then(function() { //<<<< here's an additional `.then()` chained to `when.all(...)`
        return out; //<<<<< `return` here makes the populated `out` available to the `.done()` callback below.
    });
})
.catch(console.log)
.done(function (out_) {
    console.log(out_);
});

The ugly outer member has disappeared!

In the .done() callback, I have changed the member name to out_ in order to emphasize that it is passed as a consequence of that return out, which happens only when all [geddit] the promises returned by REDISCLIENT.get() calls have successfully settled.

fantastic - thanks very much for the detailed breakdown - really helpful, especially the understanding how `out` is passed by return. Had me very confused! — Ben, Jul 20 '15 at 17:21
Yes, it's not exactly obvious that some nested return value should pop out several lines later as some other function's argument. Even having gotten used to promises, I still sometimes have to inspect other people's code very carefully to understand what's going on. — Roamer-1888, Jul 20 '15 at 18:21

Node.js/Javascript - nested promises and for loop with when.js

2 Answers2