1

Now due to the disk space issue on the Elasticsearch cluster, we need to clean up the unused indices, but we wanted to double sure that the old index is not used in any way.

We used the https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html, but it doesn't provide solid proof of whether the index was used recently or not.

Best solution would be some API from ES by which we can quickly determine and delete the old indices.

Amit
  • 30,756
  • 6
  • 57
  • 88

1 Answers1

3

You can use the index stats API which allows you to see whether search or index requests have been handled by a given index since the last restart.

GET index/_stats
=>
  "_all": {
    "primaries": {
      "docs": {
        "count": 396885916,
        "deleted": 1712210
      },
      "store": {
        "size_in_bytes": 207383595268,
        "throttle_time_in_millis": 0
      },
      "indexing": {
        "index_total": 8,               <-- writes are happening if you see this increase
        "index_time_in_millis": 41,
        "index_current": 0,
        "index_failed": 0,
        "delete_total": 0,
        "delete_time_in_millis": 0,
        "delete_current": 0,
        "noop_update_total": 0,
        "is_throttled": false,
        "throttle_time_in_millis": 0
      },
      "get": {
        "total": 0,
        "time_in_millis": 0,
        "exists_total": 0,
        "exists_time_in_millis": 0,
        "missing_total": 0,
        "missing_time_in_millis": 0,
        "current": 0
      },
      "search": {
        "open_contexts": 7,
        "query_total": 30238,           <-- reads are happening if you see this increase
        "query_time_in_millis": 254000,
        "query_current": 0,
        "fetch_total": 816,
        "fetch_time_in_millis": 25997,
        "fetch_current": 0,
        "scroll_total": 7637,
        "scroll_time_in_millis": 686963111,
        "scroll_current": 7,
        "suggest_total": 0,
        "suggest_time_in_millis": 0,
        "suggest_current": 0
      },
Val
  • 207,596
  • 13
  • 358
  • 360
  • I used this api but as I would hit this api in a script which would list me all the unused indices, and to confirm index is being used or not, I need to hit it multiple times in various interval also need to take care that its not during the off peak time to avoid the case, when there is no traffic. – Amit Feb 21 '19 at 07:43
  • then you already have all you need, just let that script run and report over a 24 hours period. Run it once at the beginning, and once at the end, subtract, conclude. This is the same API that Marvel (now called XPack Monitoring) uses. – Val Feb 21 '19 at 07:54