0

I'm implementing a logger database using MongoDB. The capped collection will contain log messages collected from several sources across the network. Since I want to do $lte/$gte queries on _id afterwards I need to have an _id that grows as a monotonic function.

To achieve that I've implemented the auto-incremented counter described in this article http://docs.mongodb.org/manual/tutorial/create-an-auto-incrementing-field/

My code looks like that:

var mongo = require("mongodb");
var Promise = require('es6-promise').Promise;

function connectToDB(mongo, uri) {
    return new Promise(function(resolve, reject) {
        mongo.MongoClient.connect(uri, function (err, db) {
            if(err) reject(err);
            else resolve(db);
        });
    });
}

function getNextSequenceNumber(db, counterName) {
    return new Promise(function(resolve, reject) {
        db.collection("counters", function (err, collection) {
            if (err) reject(err);
            else {
                var criteria = { _id: counterName };
                var sort =  {$natural: 1};
                var update =  { $inc: { seq: 1 } };
                var options = {remove: false, new: true, upsert: true};
                collection.findAndModify(criteria, sort, update, options, function(err, res) {
                    if(err) reject(err);
                    else resolve(res.seq);
                });
            }
        });
    });
}

It works perfectly fine, but I've read that by default the number fields used in MongoDB are actually floats. The problem is that my database is a capped collection of log entries and it is going to have lots of entries. Moreover, since this is a capped collection the old entries will be overwritten but the counter will keep growing. Having counter as a float I cannot guarantee the system will keep on working after a few years. My question is how can I force MongoDB to use 64 bit counter in this particular case.

Please provide some code examples.

conceptacid
  • 248
  • 2
  • 13
  • 2
    As an aside, this sounds like a bad idea for the scenario you state. The ObjectId is really good enough to range on by default and I would try and avoid this on a collection that you describe. – Sammaye Jul 21 '15 at 21:07
  • Well, I don't get the point. Your log entries will have a timestamp, anyway, right? Why don't you go with the ObjectId for the log entries and create an index over the timestamp and do your calculations there? Furthermore, I do not understand the use of a capped collection the first place. Just for making eviction sure, a [TTL index](http://docs.mongodb.org/manual/tutorial/expire-data/) would be by far more elegant and flexible in terms of scalability: You could have billions of log entries and they *still* would be removed when older than say a month. – Markus W Mahlberg Jul 21 '15 at 22:15
  • @Sammaye, thanks for pointing this out. I'm still considering to use it. One thing against it was as documentation says _The relationship between the order of ObjectId values and generation time is not strict within a single second. If multiple systems, or multiple processes or threads on a single system generate values, within a single second;_ – conceptacid Jul 22 '15 at 07:00
  • @MarkusWMahlberg, I can't use the server timestamp because the events can come on the same millisecond. If I wanted to do this I must add up some counter anyway to make it unique. Regarding the TTL index, I think you are right - this is probably a better solution than a capped collection. – conceptacid Jul 22 '15 at 07:22
  • @conceptacid I solved this by using the ObjectId as the definitive time. I had the same problem on a IM app whereby messages where sent 10's in a second. I used the ObjectId for everything there and considered it authoritative for time which hasn't let me down yet. It does mean, as you quote, that ObjectIds can be before one or the other in the same second but the user will never notice, plus the increment of single second does mean there is still some order within that second. – Sammaye Jul 22 '15 at 08:51
  • @conceptacid I was talking of using *both* `ObjectId` (which will give you the uniqueness *and* the log entries native timestamp (which you need since it is very likely that things happen the same time). Maybe I don't get your use case, and it might make sense to elaborate it a bit. – Markus W Mahlberg Jul 22 '15 at 12:35
  • @Sammaye yes, I agree – conceptacid Jul 22 '15 at 14:50
  • @MarkusWMahlberg In the end I decided to drop the counter idea and use ObjectId, thanks for the input! – conceptacid Jul 22 '15 at 14:51

1 Answers1

2

MongoDB (or rather BSON) has the NumberLong type, which is a 64-bit signed integer.

From Node.js you can use it in your update statement to create the seq property of that type:

var update =  { $inc : { seq : mongo.Long(1) } };

This also seems to convert the seq property of existing documents to NumberLong.

robertklep
  • 198,204
  • 35
  • 394
  • 381