1

I am using Elasticsearch for storing microservices logs. All microservices log in common patterns and by Fluentd logs collected and shipped to index name pattern like log-${serviceName}-%Y.%m.%d.

I defined an index template for log-- and create an ILM policy to rollover indices to the delete phase after 2 days and delete them after 4 days. and connect the ILM policy to the index-template with my-log-alias.

So I need something like this: each day, there are for example 10 active indices that log documents written to them. and after 2 days these indices all go to the delete phase.

  1. Can I use one index template and one ILM policy for all of my services?
  2. And What's wrong with my setting on elasticsearch index-template and policy?
  3. Am I using this feature in the right way?

Thank you for reading.

index-template:

{
  "order": 0,
  "index_patterns": [
    "log-*-*"
  ],
  "settings": {
    "index": {
      "lifecycle": {
        "name": "my-log",
        "rollover_alias": "my-log-alias"
      },
      "number_of_replicas": "1"
    }
  },
  "aliases": {
    "sb-log": {}
  },
  "mappings": {
    "_doc": {
      "properties": {
        "level": {
          "ignore_above": 256,
          "type": "keyword"
        },
        "message": {
          "type": "text"
        }
      }
    }
  }
}

ilm-policy

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "2d",
            "max_size": "50gb"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "4d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
Ali Malek
  • 578
  • 8
  • 26

1 Answers1

-1
  1. Can I use one index template and one ILM policy for all of my services?

Yes, you can.

  1. And What's wrong with my setting on elasticsearch index-template and policy?

With your definition, the indexes are rolloved after 2 days (or 50gb) and, after 4 days from the rollover action, will be deleted.

Gianluca Pinto
  • 235
  • 3
  • 6
  • It's not working because of defining the rollover_alias, it seems the ILM, don't automatically work with a dynamic alias for each index. – Ali Malek Dec 11 '20 at 16:29
  • but are you applied the policy to the index template? https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html#apply-policy-template – Gianluca Pinto Dec 14 '20 at 08:33
  • Yes, I applied it to the index template. – Ali Malek Dec 14 '20 at 09:20