0

I'm working on a node.js project. I'm trying to understand how MongoDB works. I'm obtaining data hourly via a cron file. I'd like for there to be unique data, so I'm using update instead of insert. That works fine. I'd like to add the option that the data expires after three days. Its not clear to me how to do that.

In pseudo code:

Setup Vars, URL's, a couple of global variables, lineNr=1, end_index=# including databaseUrl. 

   MongoClient.connect(databaseUrl, function(err, db) {
    assert.equal(null, err, "Database Connection Troubles: " + err);
 **** db.collection('XYZ_Collection').createIndex({"createdAt": 1},
            {expireAfterSeconds: 120}, function() {});   **** (update)

   s = fs.createReadStream(text_file_directory + 'master_index.txt')
        .pipe(es.split())
        .pipe(es.mapSync(function(line) { 
                s.pause();  // pause the readstream
                lineNr += 1;
                getContentFunction(line, s); 
                if (lineNr > end_index) {
                    s.end();
                }
        })
        .on('error', function() {
            console.log('Error while reading file.');
        })
        .on('end', function() {
            console.log('All done!');
        })
    );

    function getContentFunction(line, stream){ 
        (get content, format it, store it as flat JSON CleanedUpContent)
        var go = InsertContentToDB(db, CleanedUpContent, function() {
            stream.resume();
        });
    }  

    function InsertContentToDB(db, data, callback)   
            (expiration TTL code if placed here generates errors too..)             
            db.collection('XYZ_collection').update({
                    'ABC': data.abc,
                    'DEF': data.def)
                }, {
                    "createdAt": new Date(),
                    'ABC': data.abc,
                    'DEF': data.def,
                    'Content': data.blah_blah
                }, {
                    upsert: true
                },
                function(err, results) {
                    assert.equal(null, err, "MongoDB Troubles: " + err);
                    callback();
                });
    }

So the db.collection('').update() with two fields forms a compound index to ensure the data is unique. upsert = true allows for insertion or updates as appropriate. My data varies greatly. Some content is unique, other content is an update of prior submission. I think I have this unique insert or update function working correctly. Info from... and here

What I'd really like to add is an automatic expiration to the documents within the collection. I see lots of content, but I'm at a loss as to how to implement it.

If I try

db.collection('XYZ_collection')
  .ensureIndex( { "createdAt": 1 }, 
                { expireAfterSeconds: 259200 } );   // three days

Error

/opt/rh/nodejs010/root/usr/lib/node_modules/mongodb/lib/mongodb/mongo_client.js:390 throw err ^ Error: Cannot use a writeConcern without a provided callback at Db.ensureIndex (/opt/rh/nodejs010/root/usr/lib/node_modules/mongodb/lib/mongodb/db.js:1237:11) at Collection.ensureIndex (/opt/rh/nodejs010/root/usr/lib/node_modules/mongodb/lib/mongodb/collection.js:1037:11) at tempPrice (/var/lib/openshift/56d567467628e1717b000023/app-root/runtime/repo/get_options_prices.js:57:37) at /opt/rh/nodejs010/root/usr/lib/node_modules/mongodb/lib/mongodb/mongo_client.js:387:15 at process._tickCallback (node.js:442:13)

If I try to use createIndex I get this error...

`TypeError: Cannot call method 'createIndex' of undefined`

Note the database is totally empty, via db.XYZ_collection.drop() So yeah, I'm new to the Mongo stuff. Anybody understand what I need to do? One note, I'm very confused by something I read: in regards to you can't create TTL index if indexed field is already in use by another index. I think I'm okay, but its not clear to me.

There are some restrictions on choosing TTL Index: you can't create TTL index if indexed field is already used in another index. index can't have multiple fields. indexed field should be a Date bson type

As always, many thanks for your help.

Update: I've added the createIndex code above. With an empty callback, it runs without error, but the TTL system fails to remove entries at all, sigh.

Community
  • 1
  • 1
zipzit
  • 3,778
  • 4
  • 35
  • 63
  • It would appear that you have not wrapped that call in similar function to the others and/or are not calling it by passing in the `db` variable value. You are also notably missing a "callback" on the `.ensureIndex()` call ( should be using `.createIndex()` really ) though that is not the current problem. We are missing the context for your new piece of code, though it's easy to tell the other code was not written by yourself. – Blakes Seven Mar 08 '16 at 06:13
  • BTW. TTL should be sufficient here, but your referenced passage has nothing to do with the error. The error is caused in the context of code you have not listed here, and how you are calling this. – Blakes Seven Mar 08 '16 at 06:15
  • @BlakesSeven thx, for your response. So `createIndex` --> undefined. I'm not declaring the conent of the db anywhere, Its created via the MongoClient.connect() function. Will the 'undefined' error propogate if the DB collection doesn't exist anywhere? Remember I wiped the thing clean. First use of update inserts first document into a 'new' collection. Is that the root of my issue? – zipzit Mar 08 '16 at 06:22
  • Like I said, look at the other functions in your listing like `function InsertContentToDB(db, data, callback)`. The code you are quoting must be ( needs to be ) wrapped in something similar. You are not showing us that, and that is one place your problem may be. Secondly look at where calls to something like `InsertContentToDB` are being used in your code. Something is being passed in as the first `db` argument, which is also something your new code is likely missing. Again we cannot see that code in your question, and that is where your problem is. – Blakes Seven Mar 08 '16 at 07:03
  • So I added a simple empty callback and the error messages disappear. I'm testing right now. One thing. Is the createIndex message posted once per document or once per collection? I'm guessing its once per collection but its not clear to me. My actual code is `db.collection('XYZ_collection').createIndex({ "createdAt": 1 }, { expireAfterSeconds: 120 }, function() {});` but it appears to not be working. Will Mongo log confirm the TTL system is alive? – zipzit Mar 08 '16 at 07:15
  • My bad. I read [this article](http://gavruk.blogspot.com/2012/09/mongodb-time-to-live-ttl-collections.html) again a few more times. He says up to a minute after expiration for items to die, and setting the collection should catch documents inserted prior to the CreateIndex() This implies the function call it only needed once per collection... My bad. Oh, and what is a `note: noprealloc may hurt performance in many applications` log entry about? – zipzit Mar 08 '16 at 07:22

0 Answers0