0

I am facing this issue with mongodb. My code is something like this

for(loop) {
    var cursorQuery = db.beacon_0000.aggregate([
        {
            $match: {
                ...
            }
        },
        {
            $project: {
                ...
            }
        },
        {
            $group: {
                ...
            }
        },
        {
            $sort: {
                ...
            }    
        }
    ], {allowDiskUse: true} );
     ...
    while(cursorQuery.hasNext()) {
        var cursor = cursorQuery.next();
        ...
    }
}

I run the above query via command and mongo shell as

$ mongo dbName file.js

After a while I get the cursor didn't exist on server error at line cursorQuery.hasNext().

In find query if I get this error, I can resolve by adding addOption(DBQuery.Option.noTimeout) However this option does not seem to be available with aggregate

Please let me know how can I resolve or workaround this issue.

Just to provide additional update: When say I use

var cursor = db.collection..aggregate([ ...], {allowDiskUse: true} ).addOption(DBQuery.Option.noTimeout)

I get this error E QUERY TypeError: Object # has no method 'addOption'

However when say I use

var cursor = db.collection..find({...}, {...}).addOption(DBQuery.Option.noTimeout)

It works fine.

Checking the aggregate doc https://docs.mongodb.com/v3.0/reference/method/db.collection.aggregate/ It says: Returns:A cursor to the documents produced by the final stage of the aggregation pipeline operation

And then checking cursor doc https://docs.mongodb.com/v3.0/reference/method/cursor.addOption/#cursor.addOption

There is no suggestion that aggregate cursor is different from find cursor and former does not support DBQuery.Option.noTimeout.

So is there a bug at mongodb for this. Any way to fix it or have a workaround.

Note mongodb version is 3.0

Sachin Mittal
  • 98
  • 2
  • 8

3 Answers3

1

I had the same issue and solved it by setting the idle cursor timeout from default 10 minutes to 1 hour. This is configurable since mongodb 2.6.9. See:

The default cursor timeout is 600000 ms = 10 minutes. You can alter it in different ways:

  • on startup: mongod --setParameter cursorTimeoutMillis=<num>
    • or: mongos --setParameter cursorTimeoutMillis=<num>
  • or during operation, using the mongo shell: db.adminCommand({setParameter:1, cursorTimeoutMillis: <num>})

Mongos is not transferring the command to its mongod's of the cluster. Also the Primary does not replicate the command to its replicaSet members. Thus, you need to execute the command on every mongos and mongod where the query might run.

Kay
  • 624
  • 1
  • 7
  • 17
  • cursorTimeoutMillis changes the default timeout period server-wide. If this is not possible, for the time being the solution is recoding aggregates so they run faster. – Corral Apr 18 '17 at 16:33
0

You have sort of answered this yourself.

Adding the option addOption(DBQuery.Option.noTimeout) will indeed fix the issue when using find because it stops the cursor from timing out and therefore it will exist when you try .hasNext()

However the cursor for aggregation does not have that option so you can't stop it from timing out unfortunately.

Andy Jenkins
  • 617
  • 8
  • 26
  • So what are my options here. Would setting a batchSize to a large number help here. say db.collection.aggregate().batchSize(10000) Anything else you can suggest to prevent cursor from timing out for aggregate pipelines. – Sachin Mittal Aug 04 '16 at 13:07
  • Someone that makes the aggregation take less time is what you want. Considering you can't prevent the `cursor` from timing out you need to use it within the default timeout length. It does look like they are trying to fix this though. See: https://jira.mongodb.org/browse/SERVER-6036 – Andy Jenkins Aug 04 '16 at 13:11
0

you can actually use : maxTimeMS
as it described in documentation:

Optional. Specifies a time limit in milliseconds for processing operations on a cursor. If you do not specify a value for maxTimeMS, operations will not time out. A value of 0 explicitly specifies the default unbounded behavior.

There is an option in mongodb documentation that you can set it no a non-negative number for the time that you want your cursor to be alive. you can see the more detail on : documentation

Samin Fakharian
  • 63
  • 1
  • 10