How to get cursor.forEach() iterator to wait for promise chain to resolve?

Question

Using node.js mongodb native module I'm trying to use a promisified iterator but it is not waiting for the promise chain to resolve before firing the finalCallback

db.collection('items').find({}).forEach(promiseIterator, finalCallback);

function promiseIterator(doc){
    return Promise.resolve(doc)
        .then(res => {
            console.log(res); // this fires *after* finalCallback fires
        })
}

function finalCallback(res){
    console.log(res); // this fires *before* promiseIterator resolves all promise chains
}

The doc is here: https://mongodb.github.io/node-mongodb-native/2.2/api/Cursor.html#~resultCallback

Their ES6 examples all use generators so I'm not sure if Promises even work here. But the promise chain is not fully resolved before firing finalCallback.

https://mongodb.github.io/node-mongodb-native/2.2/reference/ecmascript6/crud/

The reason I'm having this issue is because my promiseIterator needs to make several async/promisified calls on each document. (I am unable to load all docs into memory with .toArray() as there are over a million documents and I get a process out of memory error when I try that.

antti.mann · Answer 1 · 2016-12-17T14:08:05.847

1

Unfortunately you cannot. Mongodb native's forEach doesn't wait for Promises to resolve as you noticed. They just call for a callback function and continues to next object.

Cursor's forEach source code (line 769)

http://mongodb.github.io/node-mongodb-native/2.2/api/lib_cursor.js.html

You need to iterate manually with next and hasNext.

http://mongodb.github.io/node-mongodb-native/2.2/api/Cursor.html#next

EDIT: Example added.

You can use promises exactly as jstice4all succested. Here is his example with more details.

db.collection('items').find({}, (err, resultCursor) => {
  resultCursor.next(processItem);

  function processItem(err, item) {
    if(item === null) return; // All done!

    // Add your asynchronous function/chain of functions in place of Promise.resolve()
    Promise.resolve().then(() => {
      resultCursor.next(processItem); // Read next item from database
    });
  }
});

Of course this will process all objects serially, one after another. If you want to process all objects in parallel and wait for them to finish before doing something else, you should use collect all created promises with Promise.all.

btw. nextObject is deprecated, so you can use next instead of it.

edited Dec 17 '16 at 14:08

answered Dec 14 '16 at 15:09

antti.mann

11
3

Can you provide example? I tried to use `yield` from their docs but I get a syntax error. I'm using node v7. Their docs don't have any Promise examples (I only see generators) yet they claim you can use them. – chovy Dec 14 '16 at 19:15
Where is the `externalAsyncFunction()` called from in your example? – chovy Dec 14 '16 at 22:56
My example was just referencing jstice4all's example which seems to be correct. Better refer that than copy example in thia – antti.mann Dec 15 '16 at 07:02
I do not get a `next` callback passed to `externalAsyncFunction` – chovy Dec 17 '16 at 05:30
I combined jstice4all's example with my example. It should be clear now. – antti.mann Dec 17 '16 at 14:10

score 1 · Answer 2 · answered Dec 14 '16 at 15:19

1

You can use a recursion with nextObject instead of forEach:

db.collection('items').find({}, function(err, resultCursor) {
  function processItem(err, item) {
    if(item === null) {
      return; // All done!
    }

    externalAsyncFunction(item, function(err) {
      resultCursor.nextObject(processItem);
    });

  }

  resultCursor.nextObject(processItem);
}

answered Dec 14 '16 at 15:19

jstice4all

1,878
4
22
33

Is it possible to use promises here with `nextObject`? – chovy Dec 14 '16 at 19:15
Yes, just call `nextObject` from `then`. – jstice4all Dec 14 '16 at 20:54
Would you mind adding an example that uses promises with `.find()` and cursor instead of callbacks. I think this is the correct answer but I want to use promises. – chovy Dec 14 '16 at 20:59

How to get cursor.forEach() iterator to wait for promise chain to resolve?

2 Answers2