Delete documents older than 30 days in elasticsearch

Question

I want to delete documents in my elasticsearch index which are older than 30 days.

Any ideas?

EDIT:

I want this to happen automatically - no document in my index shoudl be older than 30 days. So, in my opinion there are 2 options: either using curator or DELETE requests.

I have tried both, but i failed. Somehow i have to create a filter which filters all documents older than 30 days and deletes them, when i am using DELETE http statement.

I tried with curator, but curator (as far as i understood this) deletes only whole indices. When attempting to delete indices older than 30 days with curator, my timestamp causing errors.My moment.js pattern looks like this"MMMM Do YYYY, HH:mm:ss.SSS".

EDIT 2: I added the following to my logstash configuration:

elasticsearch
    {
    hosts => ["http://localhost:9200"]
    index => "logstash-%{type}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
    }

So logstash creates for every type and every day a particular index. Now i can use curator to delete the indices older than a specific date.

Problem solved imho.

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html `POST twitter/_delete_by_query { "query": { { "range": { "@timestamp": { "gte": 1548975600000, "format": "epoch_millis" } } } } }` and calculate the epoch in your script. — sastorsl, May 04 '19 at 09:34

score 30 · Answer 1 · edited Mar 10 '19 at 09:56

30

You can use DELETE query for that: https://www.elastic.co/guide/en/elasticsearch/reference/1.6/docs-delete-by-query.html in example the query will delete everything older than: 2016-02-29

DELETE index_name/_query
{
  "query": {
    "filtered": {
      "query": {
        "query_string": {
          "query": "*"
        }
      },
      "filter": {
        "range": {
          "@timestamp": {
            "lte": "2016-02-29"
          }
        }
      }
    }
  }
}

Update >6.4

According to the official documentation, this function has been deprecated and replaced by _delete_by_query

POST index_name/_delete_by_query
{
  "query": { 
    "match": {
      "message": "some message"
    }
  }
}

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html

edited Mar 10 '19 at 09:56

Mr. bug

366
2
11

answered Mar 29 '16 at 15:24

Кирилл Полищук

845
6
4

what is @timestamp, should I create this field when creating a new index, because in 6.2 elasticksearch I can not find it. – Palaniichuk Dmytro Mar 29 '18 at 14:36
3

This unfortunately isn't supported anymore. As of 6.4 use POST _delete_by_query https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html – Dimitri W Sep 19 '18 at 20:12

Delete documents older than 30 days in elasticsearch

1 Answers1