2

I m actually facing a problem when trying to get a full resultset (~40 result). Each object has ~260 childs references and other objects packed in.

Actually, when I try to display 30 objects, it works. But when I try with 40 it doesn't. Each object is correct, no data is invalid.

It seems that I've exceed the maximum size(4 or 16 mb) authorized by mongoDb or defaultly Mongoskin (the library that I use to access mongodb), and mongodb close the pool connection to my application.

So how can I increase the maximum document size, or the maximum result set size, to display everything I need ?

EDIT : My collection size is 8Mb (from MongoVue informations)

EDIT 2 : The code I use to get my data :

db.collection("roadmap").find({}).toArray()

An screenshot of my db : enter image description here

EDIT 3 : I changed the way I was exploiting my data using 

@em.collection('roadmap').find({}, (err, resultCursor)=>
      nextItem = (err, item)=>
        if !item)
          return

        console.log item
        resultCursor.nextObject(nextItem)
      resultCursor.nextObject(nextItem)
    )

It doesn't work neither, the request is executed, and then, I lose my mongodb connection. If I comment this piece of code in my application, all is working great.

EDIT 4 : Full stack trace, it's more a library problem than a db problem

Error: parseError occured
  at [object Object].<anonymous> (/var/www/xxxxxx/Ws/node_modules/mongoskin/node_modules/mongodb/lib/mongodb/connection/connection_pool.js:192:34)
  at [object Object].emit (events.js:98:17)
  at Socket.<anonymous> (/var/www/xxxxxx/Ws/node_modules/mongoskin/node_modules/mongodb/lib/mongodb/connection/connection.js:393:20)
  at Socket.emit (events.js:95:17)
  at Socket.<anonymous> (_stream_readable.js:764:14)
  at Socket.emit (events.js:92:17)
  at emitReadable_ (_stream_readable.js:426:10)
  at emitReadable (_stream_readable.js:422:5)
  at readableAddChunk (_stream_readable.js:165:9)
  at Socket.Readable.push (_stream_readable.js:127:10)
mfrachet
  • 8,772
  • 17
  • 55
  • 110
  • Can you post the query that's giving you the problem, an example document (abbreviated is fine), and the exact error messages that you get? Also, MongoDB and driver versions are helpful. – wdberkeley Dec 12 '14 at 16:54
  • I added the code and a screenshot :-) – mfrachet Dec 15 '14 at 07:48
  • I think the problem is `.toArray()`, not the `find`. But I don't see the exact error message, so it's hard to tell what the problem is. – wdberkeley Dec 15 '14 at 20:54
  • I dont have anything else than "Parse error" in the log console :-/ – mfrachet Dec 16 '14 at 07:46
  • And I m using https://github.com/kissjs/node-mongoskin . What is the alternative of toArray ? – mfrachet Dec 16 '14 at 09:23
  • Can it be that since it is a big result set, the cursor expires? I think that you can set a timeout time to the cursor. http://docs.mongodb.org/manual/core/cursors/ – leojg Dec 16 '14 at 15:52
  • Can you try the same thing from the mongo shell? 40 documents, even with 260 fields, is not much data. – wdberkeley Dec 16 '14 at 16:54
  • http://stackoverflow.com/questions/19715229/is-there-a-mongodb-maximum-bson-size-work-around – Disposer Dec 17 '14 at 06:50
  • @wdberkeley, it works in the mongoshell – mfrachet Dec 17 '14 at 07:16
  • @Disposer, I read your post, but it's apparently not a size problem with mongo documents, and I cant redesign my DB actually – mfrachet Dec 17 '14 at 07:17
  • @leojg it seems that the problem occurs on the 25 (no matter the object) but changing the timeout doesn't seem to work :/. In fact, I can batchsize only to 25, after, I got the error. – mfrachet Dec 17 '14 at 08:41
  • @Skahrz Looking at your stack trace I think that the problem might be that you have some non-utf8 char in some of the documents. – leojg Dec 17 '14 at 11:33
  • I will check. But I can see the whole result using skip and limit. It s just when I want to get the full resultset in one time that I ve the error – mfrachet Dec 17 '14 at 11:37

2 Answers2

1

I would suggest you to Create and index for the collection 'roadmap'. 

 db.roadmap.ensureIndex({zone:1});

This is just used to retrive data faster than before, refer http://docs.mongodb.org/manual/core/indexes-introduction/

Change the Find as follows, there is no need of toArray function since the data is already an array.

 db.roadmap.find({},function(err,data){
 if(err){
 console.log(err);
}
console.log(data); 
)

This finds all the data in the roadmap collection.The index is used when the data set is huge. Hope this helps.

SUNDARRAJAN K
  • 2,237
  • 2
  • 22
  • 38
1

That error is being emitted by the mongo node driver "below" mongoskin. In the connection file, there are several places that can emit this error. To see the actual error, give the connection a logger and it will print the thing. That will tell you where in the connection you have a problem, and what it is. Note that mongoskin is using version 1.4.4 of the driver, so if it is a bug they might have fixed it.

Spencer Rathbun
  • 14,510
  • 6
  • 54
  • 73
  • Thank you for this answer. I m actually using a version of mongoskin using the 1.4.4 mongodriver. I m actually looking for a method to give a logger to mongodb driver through mongoskin to give your more information – mfrachet Dec 17 '14 at 13:44
  • You'll probably need to reach into the guts of the connection object after you create it and add the property. – Spencer Rathbun Dec 17 '14 at 13:48
  • Okay, so I fond that the error is here : https://github.com/mongodb/node-mongodb-native/blob/V1.4.4/lib/mongodb/connection/connection.js#l386-395 – mfrachet Dec 17 '14 at 14:09