In Couchbase, expired document included to query view list with NULL contents

Question

Recently we started to use couchbase, We are using java spring-data-couchbase with Jersey to access couchbase. Accessing low level java-sdk-api we set expire time (TTL) to a particular document with the KEY(id). It's working fine. The code is as follows.

// define couchbaseTemplate for lower-level access to Java SDK
@Autowired
CouchbaseTemplate couchbaseTemplate;


// setExpiry method update expiry given a doc ID
@Override
public void setExpiry(String key, int expN) throws RepositoryException {
    couchbaseTemplate.getCouchbaseClient().touch(key, expN);
}

The problem we face is when we try to get list of documents using query, the list contains the expired documents. And when we try to access the documents from the list we found it to be null.

But if we execute query after a while the expired document no longer include to the list.

Example: When the expN = 10 seconds, and we execute query around 10 seconds after setting the TTL, the expired documents included If we execute query around 20 seconds after setting the TTL, the expired documents no longer included

in stale options we set

Query.setStale(Stale.false)

We have tried to manipulate

Query.setIncludeDocs

But no luck, any help....

Paddy · Answer 1 · 2015-07-29T12:50:00.487

Couchbase Server does expiries lazily. There are three ways an item can be expired:

When a document is accessed (get operation) the expiration value is checked
When the expiry pager runs
When disk compaction process runs (Only in Couchbase Server 3 and onwards)

As a result of this views will not be updated until one of these three processes has happened.

For this use case you could simple do a range query against the view using the current time so it only returns documents that have not expired. Assuming the time is the same on the cluster are well as the client and the view being used is this one:

function (doc, meta) {
  emit(meta.expiration, null);
}

The meta.expiration is an epoch timestamp, so the following query could be used:

String currentEpoch = String.valueOf((System.currentTimeMillis()/1000)); 
bucket.query(ViewQuery.from("designdoc", "myview").startkey(currentEpoch));

Please note that this will return all alive documents that have an expiration set.

If you want to do something more interesting with date formats have a look at the Date and time selection half way down in the View and query examples chapter in the Couchbase Server manual.

score 0 · Answer 2 · edited Jun 20 '20 at 09:12

As the Couchbase official documents said,

Detecting Expired Documents in Result Sets : If you are using views for indexing items from Couchbase Server, items that have not yet been removed as part of the expiry pager maintenance process will be part of a result set returned by querying the view. To exclude these items from a result set you should use query parameter include_doc set to true.

For expired documents, if you set include_doc=true , Couchbase Server returns a result set indicating the document does not exist anymore. Specifically, the key that had expired but had not yet been removed by the cleanup process will appear in the result set as a row where "doc":null :

So, this is how Couchbase works with expired documents.

For your case, just filter out the items where the doc is null, the rest will be your expected result.

In Couchbase, expired document included to query view list with NULL contents

2 Answers2