1

We have approx 1 million keys in CB and each key has expiry time of 10 days. Basically after every 10 days we run a schedular to re-insert data in CB for these 1 million keys.

Out of these there are approx 0.5 million keys which are not used. There is no way to identify which keys will be used and which will not.

I have checked if there is a way to define 2 TTL for a key.

  1. Delete record after fixed certain time period. (expiry)
  2. Delete record if key not used for certain time period

Is there some way to delete these unused keys.

Vipin Gupta
  • 213
  • 3
  • 15

2 Answers2

3

This is something that is easily implemented in the application tier. Say you wanted all new keys to be deleted after 90 days and delete any keys that are unused for 30 days.

When you create the document, add a creation timestamp field and a TTL of 30 days. Then when your application reads/updates a document, you can set a new TTL on the document which is calculated as the timestamp + 90 days.

After 30 days any unused docs will be expired, documents which have been accessed and updated will have a TTL in 60 additional days.

Thanks, Ian McCloy (Principal Product Manager, Couchbase)

Ian McCloy
  • 306
  • 3
  • 2
1

The Couchbase Eventing service can be periodically run to remove old data, it doesn't need an index so in some situations it is good solution. This isn't as clean as Ian's TTL approach but works if you have a mixed set of documents some you want to purge and other you want to keep.

Refer to this post: https://forums.couchbase.com/t/delete-1-billion-doc-having-docs-key-company-name-employeeid/27261/4

Note in later versions of Couchbase you can also pass a constant into the Eventing function for the "cutoff_millis" timestamp or just calculate it from Date.now().

If you wanted to automate this you could deploy and undeploy via the Eventing REST API and a crontab job.

Jon Strabala
  • 421
  • 3
  • 3