MongoDB: is it possible to capture TTL events with Change Stream to emulate a scheduler (cronjob)?

Question

I'm new to MongoDB and I'm looking for a way to do the following:

I have a collection of a number of available "things" to be used. The user can "save" a "thing" and decrement the number of available things. But he has a time to use it before it expires. If it expires, the thing has to go back to the collection, incrementing it again.

It would be ideal if there was a way to monitor "expiring dates" in Mongo. But in my searches I've only found a TTL (time to live) for automatically deleting entire documents.

However, what I need is the "event" of the expiration... Than I was wondering if it would be possible to capture this event with Change Streams. Then I could use the event to increment "things" again.

Is it possible or not? Or would there be a better way of doing what I want?

Answer 1

I was able to use Change Streams and TTL to emulate a cronjob.

But, basically, anytime I need to schedule an "event" for a document, when I'm creating the document I also create an event document in parallel. This event document will have as its _id the same id of the first document.

Also, for this event document I will set a TTL.

When the TTL expires I will capture its "delete" change with Change Streams. And then I'll use the documentKey of the change (since it's the same id as the document I want to trigger) to find the target document in the first collection, and do anything I want with the document.

I'm using Node.js with Express and Mongoose to access MongoDB. Here is the relevant part to be added in the App.js:

const { ReplSet } = require('mongodb-topology-manager');

run().catch(error => console.error(error));

async function run() {
    console.log(new Date(), 'start');
    const bind_ip = 'localhost';
    // Starts a 3-node replica set on ports 31000, 31001, 31002, replica set
    // name is "rs0".
    const replSet = new ReplSet('mongod', [
        { options: { port: 31000, dbpath: `${__dirname}/data/db/31000`, bind_ip } },
        { options: { port: 31001, dbpath: `${__dirname}/data/db/31001`, bind_ip } },
        { options: { port: 31002, dbpath: `${__dirname}/data/db/31002`, bind_ip } }
    ], { replSet: 'rs0' });

    // Initialize the replica set
    await replSet.purge();
    await replSet.start();
    console.log(new Date(), 'Replica set started...');

    // Connect to the replica set
    const uri = 'mongodb://localhost:31000,localhost:31001,localhost:31002/' + 'test?replicaSet=rs0';
    await mongoose.connect(uri);
    var db = mongoose.connection;
    db.on('error', console.error.bind(console, 'connection error:'));
    db.once('open', function () {
        console.log("Connected correctly to server");
    });

    // To work around "MongoError: cannot open $changeStream for non-existent database: test" for this example
    await mongoose.connection.createCollection('test');

    // *** we will add our scheduler here *** //

    var Item = require('./models/item');
    var ItemExpiredEvent = require('./models/scheduledWithin');
    
    let deleteOps = {
      $match: {
          operationType: "delete" 
      }
    };

    ItemExpiredEvent.watch([deleteOps]).
        on('change', data => {
            // *** treat the event here *** //
            console.log(new Date(), data.documentKey);
            Item.findById(data.documentKey, function(err, item) {
                console.log(item);
            });
        });

    // The TTL set in ItemExpiredEvent will trigger the change stream handler above
    console.log(new Date(), 'Inserting item');
    Item.create({foo:"foo", bar: "bar"}, function(err, cupom) {
        ItemExpiredEvent.create({_id : item._id}, function(err, event) {
            if (err) console.log("error: " + err);
            console.log('event inserted');
        });
    });

}

And here is the code for model/ScheduledWithin:

var mongoose = require('mongoose');
var Schema = mongoose.Schema;

var ScheduledWithin = new Schema({
    _id: mongoose.Schema.Types.ObjectId,
}, {timestamps: true}); 
// timestamps: true will automatically create a "createdAt" Date field

ScheduledWithin.index({createdAt: 1}, {expireAfterSeconds: 90});

module.exports = mongoose.model('ScheduledWithin', ScheduledWithin);

Answer 2

Thanks for the detailed code.

I have two partial alternatives, just to give some ideas.

1. Given we at least get the _id back, if you only need a specific key from your deleted document, you can manually specify _id when you create it and you'll at least have this information.

2. (mongodb 4.0) A bit more involved, this method is to take advantage of the oplog history and open a watch stream at the moment of creation (if you can calculate it), via the startAtOperationTime option.

You'll need to check how far back your oplog history goes, to see if you can use this method: https://docs.mongodb.com/manual/reference/method/rs.printReplicationInfo/#rs.printReplicationInfo

Note: I'm using the mongodb library, not mongoose

// https://mongodb.github.io/node-mongodb-native/api-bson-generated/timestamp.html
const { Timestamp } = require('mongodb');

const MAX_TIME_SPENT_SINCE_CREATION = 1000 * 60 * 10; // 10mn, depends on your situation

const cursor = db.collection('items')
  .watch([{
    $match: {
      operationType: 'delete'
    }
  }]);

cursor.on('change', function(change) {
  // create another cursor, back in time
  const subCursor = db.collection('items')
    .watch([{
      $match: {
        operationType: 'insert'
      }
    }], {
      fullDocument        : 'updateLookup',
      startAtOperationTime: Timestamp.fromString(change.clusterTime - MAX_TIME_SPENT_SINCE_CREATION)
    });

  subCursor.on('change', function(creationChange) {
    // filter the insert event, until we find the creation event for our document
    if (creationChange.documentKey._id === change.documentKey._id) {
      console.log('item', JSON.stringify(creationChange.fullDocument, false, 2));
      subCursor.close();
    }
  });
});