2

I have configured my ILM to rollover when the indice size be 20GB or after passing 30 days in the hot node

enter image description here

but my indice passed 20GB and still didn't pass to the cold node

enter image description here

and when I run: GET _cat/indices?v I get:

green  open   packetbeat-7.9.2-2020.10.22-000001                         RRAnRZrrRZiihscJ3bymig  10   1   63833049            0     44.1gb           22gb

Could you tell me how to solve that please !

Knowing that in my packetbeat file configuration, I have just changed the number of shards:

setup.template.settings:
  index.number_of_shards: 10
  index.number_of_replicas: 1

when I run the command GET packetbeat-7.9.2-2020.10.22-000001/_settings I get this output:

{
  "packetbeat-7.9.2-2020.10.22-000001" : {
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "packetbeat",
          "rollover_alias" : "packetbeat-7.9.2"
        },
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "mapping" : {
          "total_fields" : {
            "limit" : "10000"
          }
        },
        "refresh_interval" : "5s",
        "number_of_shards" : "10",
        "provided_name" : "<packetbeat-7.9.2-{now/d}-000001>",
        "max_docvalue_fields_search" : "200",
        "query" : {
          "default_field" : [
            "message",
            "tags",
            "agent.ephemeral_id",
            "agent.id",
            "agent.name",
            "agent.type",
            "agent.version",
            "as.organization.name",
            "client.address",
            "client.as.organization.name",

and the output of the command GET /packetbeat-7.9.2-2020.10.22-000001/_ilm/explain is :

{
  "indices" : {
    "packetbeat-7.9.2-2020.10.22-000001" : {
      "index" : "packetbeat-7.9.2-2020.10.22-000001",
      "managed" : true,
      "policy" : "packetbeat",
      "lifecycle_date_millis" : 1603359683835,
      "age" : "15.04d",
      "phase" : "hot",
      "phase_time_millis" : 1603359684332,
      "action" : "rollover",
      "action_time_millis" : 1603360173138,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1603360173138,
      "phase_execution" : {
        "policy" : "packetbeat",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1603359683339
      }
    }
  }
}

It's weird that it's 50GB !!

Thanks for your help

abdelhalim
  • 165
  • 2
  • 12
  • Can you show the settings of your index? `GET packetbeat-7.9.2-2020.10.22-000001/_settings` ? – Val Nov 06 '20 at 10:32
  • when I run that command I get : ``` { "packetbeat-7.9.2-2020.10.22-000001" : { "settings" : { "index" : { "lifecycle" : { "name" : "packetbeat", "rollover_alias" : "packetbeat-7.9.2" }, "routing" : { "allocation" : { "include" : { "_tier_preference" : "data_content" } } }, "mapping" : { "total_fields" : { "limit" : "10000" } }, "refresh_interval" : "5s", "number_of_shards" : "10", ``` – abdelhalim Nov 06 '20 at 10:40
  • Sorry I Coudn't past all the output cause it's too long – abdelhalim Nov 06 '20 at 10:41
  • Please update your question, as it's more legible – Val Nov 06 '20 at 10:41
  • Also paste what you get from `GET /packetbeat-7.9.2-2020.10.22-000001/_ilm/explain` – Val Nov 06 '20 at 10:42
  • I updated my question, and I think there is a problem as when I run `GET /packetbeat-7.9.2-2020.10.22-000001/_ilm/explain` it's showing max_size: 50GB ! – abdelhalim Nov 06 '20 at 10:46
  • Could you tell me please how can I solve this problem ? – abdelhalim Nov 06 '20 at 13:03
  • As you can see, the max size for the rollover is 50gb (or 30 days) and not 20gb... – Val Nov 06 '20 at 13:56
  • But as it's mentioned in the first pic, in kibana it's configured to be 20gb ! :( – abdelhalim Nov 06 '20 at 14:02
  • Is it maybe because I edited it after that the index was created, so it will be 20gb only for the new index ? – abdelhalim Nov 06 '20 at 14:06
  • Can you try to stop/start the ILM `POST /_ilm/stop` and then `POST /_ilm/start`. After that run explain again to see if it picked up the new definition. – Val Nov 06 '20 at 14:27
  • unfortunately I got the same thing.. 50gb :( – abdelhalim Nov 06 '20 at 14:34
  • It's weird because when editing a policy, the updated settings are immediately taken into account even for existing indices... Are you sure you saved your policy with 20gb? – Val Nov 07 '20 at 04:50
  • yes, cause when I go to the policies on kibana and click on packetbeat, It's configured to be 20gb ! – abdelhalim Nov 07 '20 at 11:05

1 Answers1

2

So I found the solution of this problem. After updating the policy, I removed the policy from the index using it, and then added it again to those index.

abdelhalim
  • 165
  • 2
  • 12