How to encode/decode mongodb cursor?

Question

I need to build a list of "pages" so part of this there will be a cursor. The issue is that I can't find a way to encode(to string) and decode the cursor. Any idea? The Cursor interface has no "encoding" method(there is ID, though undocumented) and there is no way to create a new cursor from a string(or int).

type Cursor interface {

    // Get the ID of the cursor.
    ID() int64

    // Get the next result from the cursor.
    // Returns true if there were no errors and there is a next result.
    Next(context.Context) bool

    Decode(interface{}) error

    DecodeBytes() (bson.Reader, error)

    // Returns the error status of the cursor
    Err() error

    // Close the cursor.
    Close(context.Context) error
}

Why do I need the cursor encoded?

To provide pagination to the end client through a html or JSON API.

There is nothing to encode in the cursor. Why do you need it? — Alex Blex, Jul 03 '18 at 12:19
Check out this related / possible duplicate: [Efficient paging in MongoDB using mgo](https://stackoverflow.com/questions/40634865/efficient-paging-in-mongodb-using-mgo). — icza, Jul 03 '18 at 12:24
"there is ID, though undocumented" -- I'm confused. When I look at the Cursor doc, the first documented function is `ID()`. — Jonathan Hall, Jul 03 '18 at 13:25
@AlexBlex how are you supposed to provide pagination to the end user(i.e. via a html interface) without a "string"/encoded cursor? — themihai, Jul 04 '18 at 10:21
@icza mgo is totally different(and deprecated) than the official mongo driver — themihai, Jul 04 '18 at 10:22
@Flimzy ok, and what the "ID" of the cursor and how can you use it? What exactly it documents? — themihai, Jul 04 '18 at 10:23
@themihai Yes, I realized later. My answer below applies to the official mongo driver. — icza, Jul 04 '18 at 10:25
Although mgo is unmaintained, it doesn't mean it uses anything different than [mongodb wire protocol](https://docs.mongodb.com/manual/reference/mongodb-wire-protocol/). icza's answer apply to **any** driver built on top of it. Implementation details may differ, but the main idea is still the same - cursor is a resource. — Alex Blex, Jul 04 '18 at 10:34
@AlexBlex there is no need to lecture me about what a unmaintained project means. As I understand `minquery` requires you to maintain a separate index of integer type(ascending). — themihai, Jul 04 '18 at 10:57
@themihai Minquery does not require you to maintain a "separate" index, it requires you to base your query on an index, and it provides you means to continue listing documents from a designated index entry, from the last index entry of your previous fetch. — icza, Jul 04 '18 at 11:00
I have update the answer, if it makes it more clear why re-using cursor for pagination is a bad idea. If it doesn't stop you - go on. Maintain a pool of cursors in memory, ensure http requests are routed to the same instance, re-use the cursor. You still don't need to encode and send it to the client. — Alex Blex, Jul 04 '18 at 11:00
@AlexBlex even if I would maintain a pool of cursors in memory how could I know the page the client is trying to fetch? I would still have to provide a string representation so that the end user can store links to "pages/cursors" as it progresses through the list/pages. I've provided an answer myself and I'm planning to mark it as "accepted". — themihai, Jul 04 '18 at 12:57

themihai · Accepted Answer · 2018-07-04T12:48:14.067

MongoDB does not provide a serializable cursor. Cursor is not serializable. The recommended workaround is to use a range query and sort on a field that generally changes in a consistent direction over time such _id.

function printStudents(startValue, nPerPage) {
  let endValue = null;
  db.students.find( { _id: { $lt: startValue } } )
             .sort( { _id: -1 } )
             .limit( nPerPage )
             .forEach( student => {
               print( student.name );
               endValue = student._id;
             } );

  return endValue;
}

There is a go package minquery that tries to make the cursor query/serialization more convenient. You may find it helpful.

score 0 · Answer 2 · edited Jul 04 '18 at 10:55

A mongo.Cursor object isn't something you can encode and put away for later use, like what you intend to use it for.

A mongo.Cursor is something you use to iterate over a "live query", a stream of documents. You can't use it to return a batch of documents which you send to your client, and when the client requests more documents (the next page), you decode the stored cursor and continue where you left off. A cursor have a server side resource under the hood, which is kept for 10 minutes (configurable, see cursorTimeoutMillis) or until you close the cursor implicitly. You do not want to keep the cursor "alive" while waiting for the client if he / she needs more documents, especially in an application with large traffic. Your MongoDB would quickly run out of resources. If cursor is closed by timeout, any attempt to read from the cursor will result with error "Cursor not found, cursor id: #####"

The Cursor.Decode() method is not to decode a cursor from some encoded form. It is to decode the next document the cursor designates into a Go value.

That's why there is no magic mongo.NewCursor() or mongo.ParseCursor() or mongo.DecodeCursor() function. A mongo.Cursor is handed to you by executing queries, e.g. with Collection.Find():

func (coll *Collection) Find(ctx context.Context, filter interface{},
    opts ...findopt.Find) (Cursor, error)

How to encode/decode mongodb cursor?

2 Answers2