2

We have a custom wrapper on top of the standard elasticsearch and curator implementation at our company. I would like to know what would be the behavior of curator dealing with "Monthly/Weekly" indexes when the default "time-unit" is set to "days".

**I cannot override the default "time-unit"

Here is example format of how our monthly/weekly indexes are named

Format of Monthly Indexes

logstash-test-monthly-2018.01
logstash-test-monthly-2018.02
logstash-test-monthly-2018.03
logstash-test-monthly-2018.04
...
...
logstash-test-monthly-2018.12

Format of Weekly Indexes

logstash-test-weekly-2018.01
logstash-test-weekly-2018.02
...
...
...
logstash-test-weekly-2018.51
logstash-test-weekly-2018.52

Delete_Index.yml - Curator delete instructions

actions:
  1:
    action: delete_indices
    options:
      ignore_empty_list: true
    filters:
      - exclude: true
        filtertype: kibana
      - exclude: false
        kind: regex
        filtertype: pattern
        value: .*-monthly-.*
      - range_to: 0
        filtertype: period
        source: name
        range_from: -60
        period_type: relative
        timestring: '%Y.%m.%d'
        exclude: true
        unit: days
    description: Delete indices more than X days old
  2:
    action: delete_indices
    options:
      ignore_empty_list: true
    filters:
      - exclude: true
        filtertype: kibana
      - exclude: false
        kind: regex
        filtertype: pattern
        value: .*-weekly-.*
      - range_to: 0
        filtertype: period
        source: name
        range_from: -30
        period_type: relative
        timestring: '%Y.%m.%d'
        exclude: true
        unit: days

Implementing the above config with the monthly index retention being 60 days and weekly index retention was 30 days.

The config was executed on **April,4th,2018 and the result was**

Monthly Indexes retained after execution

logstash-test-monthly-2018.03
logstash-test-monthly-2018.04

Since the above indexes ^^ only contain 31+4=35 days of index data and not 60 days worth as expected.

I was expecting curator will retain the following indexes

logstash-test-monthly-2018.02
logstash-test-monthly-2018.03
logstash-test-monthly-2018.04

Can anyone explain why curator is unable to retain 60days worth of indexes?

kriket
  • 135
  • 7

2 Answers2

5

TL;DR: February has a shorter number of days in it, and the age calculation is a multiple of seconds * the appropriate number of units.

All of this is explained in the age filter documentation on Elastic's site.

age filter vs. period filter

The time differential means of calculation can lead to frustration.

Setting unit to months, and unit_count to 3 will actually calculate the age as 3*30*24*60*60, which is 7776000 seconds. This may be a big deal. If the date is 2017-01-01T02:30:00Z, or 1483237800 in epoch time, subtracting 7776000 seconds makes 1475461800, which is 2016-10-03T02:30:00Z. If you were to try to match monthly indices, index-2016.12, index-2016.11, 2016.10, 2016.09, etc., then both index-2016.09 and index-2016.10 will be older than the cutoff date. This may result in unintended behavior.

Another way this can cause issues is with weeks. Weekly indices may start on Sunday or Monday. The age filter’s calculation doesn’t take this into consideration, and merely tests the difference between execution time and the timestamp on the index (from any source).

Another means of selecting indices and snapshots is the period filter, which is perhaps a better choice for selecting weeks and months as it compensates for these differences.

Once you understand that the age calculation isn't more than multiplying unit_count * unit's appropriate amount of seconds, then it makes sense why retention is happening the way it is. As mentioned, you may do better with the period filter, as it works with complete days, weeks, months, and years.

untergeek
  • 863
  • 4
  • 13
0
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 2 months (based on index name), for custom-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      continue_if_exception: False
      disable_action: False
    filters:
      - filtertype: pattern
        kind: regex
        value: ^(index-pattern).*$
      - filtertype: age
        source: name
        direction: older
        timestring: "%Y.%m"
        unit: months
        unit_count: 2

This is the yml config that works for deleting the monthly indices only.

Sarthak Agrawal
  • 1