27

Is there a way to retrieve from ElasticSearch information on when a specific index was last updated? My goal is to be able to tell when it was the last time that any documents were inserted/updated/deleted in the index. If this is not possible, is there something I can add in my index modification requests that will provide this information later on?

dchar
  • 1,665
  • 2
  • 19
  • 28
  • Does this answer your question? [How to make elasticsearch add the timestamp field to every document in all indices?](https://stackoverflow.com/questions/17136138/how-to-make-elasticsearch-add-the-timestamp-field-to-every-document-in-all-indic) – Christophe Quintard Nov 05 '21 at 06:25

4 Answers4

16

You can get the modification time from the _timestamp

To make it easier to return the timestamp you can set up Elasticsearch to store it:

curl -XPUT "http://localhost:9200/myindex/mytype/_mapping" -d'
{
  "mytype": {
      "_timestamp": {
          "enabled": "true",
          "store": "yes"
      }
  }
}'

If I insert a document and then query on it I get the timestamp:

 curl -XGET 'http://localhost:9200/myindex/mytype/_search?pretty' -d '{
>  fields : ["_timestamp"],
>    "query": {
>     "query_string": { "query":"*"}
>    }
> }'
{
   "took" : 7,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
     "total" : 1,
     "max_score" : 1.0,
     "hits" : [ {
       "_index" : "myindex",
       "_type" : "mytype",
       "_id" : "1",
       "_score" : 1.0,
       "fields" : {
        "_timestamp" : 1417599223918
      }
    } ]
  }
}

updating the existing document:

curl -XPOST "http://localhost:9200/myindex/mytype/1/_update" -d'
{
  "doc" : {
      "field1": "data",
      "field2": "more data"
  },
  "doc_as_upsert" : true
}'

Re-running the previous query shows me an updated timestamp:

  "fields" : {
    "_timestamp" : 1417599620167
  }
Olly Cruickshank
  • 6,120
  • 3
  • 33
  • 30
  • Thank you Olly. I tried this and indeed the timestamp get updated along with the document. But this is only half of what I need, since I want to get the timestamp of the document that was updated most recently. Is there an easy way to do this or I have to do a query for all documents with sorted timestamp and get the top result? Also, if a document is deleted from the index, there will be no timestamp to indicate that something has changed, correct? – dchar Dec 03 '14 at 11:28
  • 2
    I would do an aggregation that returns the max _timestamp. If you need the document too, you can run a second search that uses the timestamp. You're correct on the 2nd point, if the document is deleted you won't be able to search for it. – Olly Cruickshank Dec 03 '14 at 14:25
  • I need only the timestamp. Using the max aggregation it worked like a charm. Thank you for your help! – dchar Dec 04 '14 at 08:52
  • 10
    This is deprecated in 2.0, is there any other way to get an auto generated timestamp ? – wener Jan 11 '16 at 06:34
  • Would love to find more about it. Here is one of the github threads that is tracking the same request: https://github.com/elastic/elasticsearch/issues/13462 – animageofmine Feb 25 '17 at 01:00
  • 1
    Deprecated feature, see https://stackoverflow.com/a/43521153/2017567 – Christophe Quintard Nov 05 '21 at 06:24
5

I don't know if there are people who are looking for an equivalent, but here is a workaround using shards stats for > Elasticsearch 5 users: curl XGET http://localhost:9200/_stats?level=shards

As you'll see, you have some informations per indices, commits and/or flushs that you might use to see if the indice changed (or not).

I hope it will help someone.

millenion
  • 1,218
  • 12
  • 16
4

Just looked into a solution for this problem. Recent Elasticsearch versions have a <index>/_recovery API.

This returns a list of shards and a field called stop_time_in_millis which looks like it is a timestamp for the last write to that shard.

Jilles van Gurp
  • 7,927
  • 4
  • 38
  • 46
  • 2
    I don't know why anyone would down vote this. It solves the OP's problem exactly. curl localhost:9200/my-index/_recovery?pretty | jq . | grep stop | awk '{print $NF}' | sort | tail -1 – Wayne Walker Aug 11 '21 at 00:30
  • 1
    @WayneWalker It doesn't.. shard action != document action and shard that was used for the index is not exclusive to the index. (Old) shards may still be moved (e.g. when adding an elastic node). Easiest Python solution is es.search(index='my_index', size=1, sort='my_timestamp:desc' ) – Vincent Jul 13 '22 at 11:56
  • This is incorrect. It will be reset on restart, which means unupdated index will essentially just show server uptime – XANi Mar 03 '23 at 17:06
0

A simple solution that could suffice in some use cases, could also be to look at the files that elastic search use to store its data, and sort those based on modification time, e.g.:

sudo find /var/lib/elasticsearch/ -type f -exec stat -c "%y - %n" {} ; | sort -k 1,2

This would give a conservative estimate, in the sense that data is certainly not modified later than the file with the latest timestamp.

Stephan Korsholm
  • 31
  • 4