Node+MongoDB: coll.find().toArray(cb) works for collection A, but never fires cb for collection B?

Question

UPDATE: I've narrowed this down to what appears to be a different issue, and as such have asked a separate question here.

=======

I have a mongoDB instance running on localhost with two collections, "mydocs" (which has ~12,000 documents in it) and "mydoctypes" (which has only 7 documents in it).

I have a standalone NodeJS script which gets a connection to the database and then fires off the following:

myDb.collection('mydoctypes').find().toArray(function(err, results) {
    console.log("Got results.");
    if (err) {
        console.log("err: " + err);
    } else {
        console.log("Got doctypes: " + results.length); 
    }
});

The output of that script is:

Got results.
Got doctypes: 7

If I modify the same script to access the 'mydocs' collection instead:

myDb.collection('mydocs').find().toArray(function(err, results) {
    console.log("Got results.");
    if (err) {
        console.log("err: " + err);
    } else {
        console.log("Got docs: " + results.length); 
    }
});

I get no output at all. The callback, apparently, never gets fired.

== UPDATE ==

So it looks like the problem was likely too many documents causing toArray() to run out of RAM.

Now, I'm using .each() to iterate, but having a different issue: each() is only running through the first batch (whatever I set batchSize to), and never loading any more documents. The code is this:

myDb.collection('mydocs').find().batchSize(50).each(function(err, item) {
    if (item != null) {
        process.stdout.write(".");
    }
}

Whats the size of your docs in `mydocs`. Will they fit in RAM? — hyades, Apr 01 '16 at 19:00
@hyades It's possible that's at least part of the issue (I know toArray() would be pulling the whole collection into memory). However, I just tried myDb.collection('mydocs').find().each(function(item) { process.stdout.write("."); }) ... which ought to print out about 12k dots... instead it printed out exactly 100... does that make sense? — DanM, Apr 01 '16 at 19:09
@hyades (also... if Node was running out of RAM, wouldn't it spit out some kind of error?) — DanM, Apr 01 '16 at 19:11
Try setting a batchSize to the cursor. http://mongodb.github.io/node-mongodb-native/2.0/api/Cursor.html#batchSize — hyades, Apr 01 '16 at 19:16
@hyades Hm. .each() only iterates through the first batch, i.e. however many documents I specify in batchSize. Shouldn't each() go through the whole collection, regardless of batch size? — DanM, Apr 01 '16 at 19:23
Documents equal to `batchSize` are brought in the memory at once. It will request more once you have iterated across these. — hyades, Apr 01 '16 at 19:28
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/107986/discussion-between-danm-and-hyades). — DanM, Apr 01 '16 at 19:36

Alexandru Olaru · Answer 1 · 2016-04-02T11:29:54.323

Indeed as seen in the comments default mongodb driver for nodejs is returning a cursor, default cursor when triggered have a ~101 documents or around 1 MB batch size, you can modify this number using the batchSize function. But in order to iterate your collection you should stream it as following:

MongoClient.connect('mongodb://localhost:27017/mydb', function(err, db) {
  var cursor = db.collection('mycollection').find();
  cursor.forEach(
    function(doc) {
      console.log(doc);
    },
    function(err) {
      if (err) {
        console.error(err);
      } else {
        //cursor has exausted, no more docs to iterate exit
        return db.close();
      }
    });
});

The forEach method applied on the cursor is not the javascript default one from Arrays, it has two callbacks (the cb(doc) wich will iterate for each document, and the second one an cb(err) wich will catch the error or when the cursor was exausted.

You can use projection to lower the amount of data cursor.project({title: 1, name: 1}) and this will significantly reduce the amount of ram consumed.

Node+MongoDB: coll.find().toArray(cb) works for collection A, but never fires cb for collection B?

1 Answers1