Why sometimes Elasticsearch scroll or search returns a set of doc ids which cannot be individually retrieved?

Question

I am seeing a strange problem where Elasticsearch scroll or search API returns a set of documents which I cannot get by the ids any more. I am using Elassandra (Cassandra + ES) which is using Elasticsearch as secondary index store. There are TTL on the Cassandra records which are dropped due to TTL, but the ids are still there in Elasticsearch. Why is this strange behaviour? I did refresh and forcemerge of the corresponding index on Elasticsearch, but it didn't help.

Answer 1

Okay. I found the problem. The TTL field on Cassandra deletes the record on Cassandra, but the custom secondary index Elassandra built on Elasticsearch doesn't get deleted by that mechanism. In fact TTL is no longer there on higher version of ES. The documents need to be deleted explicitly from ES or we need to have time partioned Index on ES so that old indexes can be just deleted.