MGO TTL indexes creation to selectively delete documents

Question

I'm working with Golang and MongoDB. I have a collection which needs to keep a document which can be persistent or volatile. Hence, if it's set an expire date (as the example expireAt) the document is considered volatile and deleted otherwise it'll be kept in the collection unless it'll be manually deleted.

Reading this doc I've found an index that might work as I need it to.

Basically I need to replicate this kind of index in mgo:

db.log_events.createIndex( { "expireAt": 1 }, { expireAfterSeconds: 0 } )

db.log_events.insert( {
  "expireAt": new Date('July 22, 2013 14:00:00'),
  "logEvent": 2,
  "logMessage": "Success!"
} )

I've read (I'm searching back for the source of this information) that if the expireAt is not a valid date the deletion won't be trigger. Thus I think that all I need to do is to set the expireDate to a valid date when I need it, otherwise I'll leave it to the Go time.Time zero value.

This is my codebase

type Filter struct {
    Timestamp time.Time     `bson:"createdAt"`
    ExpireAt  time.Time     `bson:"expireAt"`
    Body      string        `bson:"body"`
}

// Create filter from data received via REST API.
var filter Filter
timestamp := time.Now()

if theUserAction == "share" { // This is action will set the document as volatile
    filter.ExpireAt = time.Now().Add(24 * time.Hour * 14)
}

filter.Timestamp = timestamp
filter.Body = "A BODY"

// Store filter in database
session, err := mdb.GetMgoSession() // This is a wrapping method that returns a valid mgo session
if err != nil {
    return NewErrorInternal("Error connecting to database", err)
}
defer session.Close()


// Get db with global data for legent
collection := session.DB(database).C(filtersCollection)

My question is: how can I set the index thus that it'll delete the document IF the expireAt key is valid? Reading the mgo documentation about Index Type it doesn't seems like there's a way to replicate the previously stated index, since the library only provides the ExpireAfter field..

Also, it's valid to assume that a zerovalue could be interpreted by mongodb as an invalid date?

From the docs it is January 1, year 1, 00:00:00.000000000 UTC which actually seems like a valid date..

What I've thought so far is doing something like this:

filtIdx := mgo.Index{
    Key:        []string{"expireAt"},
    Unique:     false,
    Background: true,
    Sparse:     false,
    ExpireAfter: 0,
}

I could be wrong, but off the top of my head I think the `expireAfterSeconds: 0` would actually be ignored with a `0` value anyway, since I "believe" it needs to be a positive value. TTL cleanup processes run at most every minute anyway, so any expiry less than 60 seconds is going to be around for at least a minute longer. As to the rest, I'm not saying you "cannot" do it, but you might want to reconsider the utility of defining an index in your Go code. Indexes only need definition once, and are therefore best implemented in "deployment scripts", rather than be part of the general code. — Neil Lunn, Apr 20 '16 at 00:48
The doc itself specify tu use `expireAfterSeconds: 0` when you want the index to expire on another date key. For what regards where the index is created, yeah, you're right it actually is in a deployment script right now. — fredmaggiowski, Apr 20 '16 at 04:50
Why would you think you are informing me something about the documentation? That is not what it says at all and the "only" key you can use is the "singular" data specified in the index. And "that" is exactly what the documentation says. Honestly, there are links to user profiles on every single comment. If you spent a moment to look, then you might understand that the person you want to "inform" has a much better concept than yourself, and was therefore "giving you some good advice". — Neil Lunn, Apr 20 '16 at 05:05
I wasn't meaning to inform you on something apart from what I honestly have understood reading it, i wasn't trying to inform you on anything (i checked out your profile too before replying) and i'm sorry if I might have expressed myself in a bad way. — fredmaggiowski, Apr 20 '16 at 05:10
By saying `the "only" key you can use is the "singular" data specified in the index` are you referring to the mgo `Index.Key` or to the `ExpireAfter`? — fredmaggiowski, Apr 20 '16 at 05:12
One field and "only" one field, which "must" be a BSON Date. And the expireAfterSeconds "must" be a positive value. Simple well documented concepts. The other point here is writing this in Go is overkill as there are far more simplistic deployment methods. So you two problems have simple solutions. 1. use a "valid" value for expiry. 2. use a simple method for index creation, rather than trying to work out how to apply to a typed language like Go. Good advice to save you lots of time. — Neil Lunn, Apr 20 '16 at 05:21
For the first point OK i'll try it out, setting my expire date to a valid date and the index expireafterseconds to a positive amount. for the second one there's nothing I can do since the indexes and databases are created by a configuration rest service written in Go.. — fredmaggiowski, Apr 20 '16 at 05:49
There's still something that's bugging me, thought... Doing this way is it still possible to make some documents to not expire at all? Since this is the behaviour I'm looking forward to obtain a collection where some documents do expire while other remain persistent — fredmaggiowski, Apr 20 '16 at 05:52

score 4 · Accepted Answer · answered Jul 27 '17 at 07:01

How can I set the index thus that it'll delete the document IF the expireAt key is valid?

An example to set a TTL index using mgo.v2 is as below:

index := mgo.Index{
    Key:         []string{"expireAt"},
    ExpireAfter: time.Second * 120, 
}
err = coll.EnsureIndex(index)

Where the above example sets to 120 seconds of expiration. See also Expire Data from Collections by Setting TTL.

Is it still possible to make some documents to not expire at all? Since this is the behaviour I'm looking forward to obtain a collection where some documents do expire while other remain persistent

You can specify omitempty flag for ExpireAt struct field as below:

type Filter struct {
    Timestamp time.Time `bson:"createdAt"`
    Body      string    `bson:"body"`
    ExpireAt  time.Time `bson:"expireAt,omitempty"`
}

Which essentially only include the field if it's not set to a zero value. See more info mgo.v2 bson.Marshal

Now, for example you can insert two documents where one would expire and the other persists. Code example:

var foo Filter
foo.Timestamp = timestamp
foo.Body = "Will be deleted per TTL index"
foo.ExpireAt = time.Now()
collection.Insert(foo)

var bar Filter
bar.Timestamp = timestamp
bar.Body = "Persists until expireAt value is set"
collection.Insert(bar)

Later on, you can set the expireAt field with an Update(), as an example:

newValue := bson.M{"$set": bson.M{"expireAt": time.Now()}}
err = collection.Update(queryFilter, newValue)

Setting a valid time value for expireAt field, would make it qualify for the TTL index. i.e. no longer persists.

Depending on your use case, alternatively you may as well Remove() the document instead of updating and relying on TTL index.

Thank you @Wan for the answer, it passed more than a year since I had this problem so right now I can't test it to verify the solution; anyway it seems to be the perfect one. I didn't though about the Gob `omitempty` keyword. The thing is that it avoids the insertion of the 0-valued Time resulting in the "deactivation" of the index for that document. Did I got it right? For now I'm upvoting the answer, I'll test it ASAP to accept it! :) — fredmaggiowski, Aug 24 '17 at 12:10

MGO TTL indexes creation to selectively delete documents

1 Answers1