How should I implement "get objects changed since" pattern with MongoDB?

Question

I have a collection of objects, let's say they are "posts," and those objects can be modified. I'd like to display a list on the client side that updates dynamically. So on the client side, if doing this via polling, the client would invoke an API like:

getPostsChangedSince(serial)

where serial could be a monotonically increasing number, probably a timestamp. The client gets back a list of posts that have changed since that time, stores a new latest-serial, and next time the client polls it requests changes since that latest serial.

I think the basic idea is the same in this question (which is about ASP.NET): How to implement "get latests changed items" with ADO.NET Data Services?

I'm trying to find the best way to implement this in MongoDB.

I like the idea of using the time for the serial, since it automatically works at least mostly correctly even if there are multiple app servers. The serial would be stored in each post object, and updated whenever the object is modified.

The timestamp-based serial could be implemented as:

• a Date (I think this is stored as a 64-bit milliseconds since epoch?)
• a Timestamp http://www.mongodb.org/display/DOCS/Timestamp+Data+Type
• something "by hand" e.g. store milliseconds as a number

Some nice features to have in a solution would include:

• ensure that creating then immediately updating an object within the OS timer resolution will still increment the serial despite it being the same time
• even better would to be guaranteed monotonic increase globally for all objects, not just guarantee that changing a given object will bump the serial on that object (absent this, getPostsChangedSince() calls probably need a fuzz backward in time, to avoid missing changes - at price of getting some changes twice)
• mongodb-side timestamps might be nice because getting the time in the app creates a gap between when you get the time, and when the new object is saved and available in queries
• update using findAndModify() with a query including the old serial, so "conflicts" (two changes at once) will throw an error allowing the app to retry

I realize some of the corner cases here are a little bit "academic" and can likely be fudged around in real life.

My approach so far is:

• use the Date type for the serial
• when modifying an object, get the current time, and if it matches the object's old serial, add 1 millisecond (yes this breaks if you make two modifications quickly without re-fetching from mongodb, but that seems OK)
• use findAndModify(), but based on https://jira.mongodb.org/browse/JAVA-276 there may not be a way to detect if it ends up not finding anything to modify (i.e. second change is ignored, in case of conflict)

Questions:

• I feel like I should use Timestamp instead; true? Any downsides?
• if you had a mongo cluster, might time in milliseconds be more unique and correct than Timestamp's time in seconds plus a number, while with one mongod Timestamp is more unique?
• is there a way to detect whether findAndModify() updated anything?
• any general advice / experiences with this problem? how would you do it?

Answer 1

You could continue to use your regular collection, as you do now, and after each update additionally insert the ID of the post into a special TTL collection. See http://docs.mongodb.org/manual/tutorial/expire-data/ for more info on using such a collection. Mongo will take care of all timing issues, you don't need to worry about serial numbers, and you can very quickly access time based lists of objects by their IDs.

Caveat:

use the blocking form of findAndModify, to ensure the changes have really been processed:

Blocking/Safe Writes

Unless you specify the "new" parameter as true the write operation will not block, and will not return an error (if there is one). If you do want the "new" document returned then the operation will wait until the write is done to return the new document, or an error.

For a "safe" (blocking) write operation you must call getLastError (if not using "new").

Answer 2

Have you considered "externalizing" the serial number generator? Time with MongoDB precision is good, but can become difficult to synchronize when involving multiple machines. One choice is that you can use memcached or something similar which is memory based, extremely fast and can be serialized (memcached has a CAS operation).

So what you would do is store a "seed" in memcached with a key say, counter. Everytime an app needs to do an insert, it gets the next number from memcached and increments the counter.

On second thoughts, you can even do away with memcached and just use a single row (sorry document) collection that just has the counter. You can get the counter and increment it which will be an extremely fast operation, mimicking memcached.

And then naturally, you can index the data appropriately. However, I am wondering that this would result in the index to be very imbalanced (right-side loped). Depending upon the situation, it might be worthwhile exploring the use of capped collection. So when you insert data into your main collection, also insert it into the capped collection and read data from that collection.