0

I've installed an Elastic Search (version 7.x) cluster and created a new index. I want to limit the maximum number of documents in this index. Let's say 10000 documents top.

The naive solution is to query the number of documents before inserting a new document into it. But this method can be not accurate and also have poor performances (2 requests...).

How to do it right?

No1Lives4Ever
  • 6,430
  • 19
  • 77
  • 140

1 Answers1

1

The best practice is to use Index Life Management which is in the Basic License and enabled by default in Elastic v7.3+

You can set a rollover action on the number of document (i put 5 max docs) :

PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_docs": 5
          }
        }
      }
    }
  }
}

Now i create a template with the policy my_policy :

PUT _template/my_template
{
  "index_patterns": [
    "my-index*"
  ],
  "settings": {
    "index.blocks.read_only" : true,
    "index.lifecycle.name": "my_policy",
    "index.lifecycle.rollover_alias": "my-index"
  }
}

Note that i put the setting "index.blocks.read_only" : true because when the rollover will be applied it will create a new index with read_only parameter.

Now i can create my index :

PUT my-index-000001
{
  "settings": {
    "index.blocks.read_only": false
  },
  "aliases": {
    "my-index": {
      "is_write_index": true
    }
  }
}

That's it ! After 5 documents, it will create a new read only index and the alias will be on writing on this one.

You can test by index some new docs with the alias : 

PUT my-index/_doc/1
{
  "field" : "value"
}

Also, by default the ilm policy will be applied every 10 minutes, you can change that in order to test with : 

PUT /_cluster/settings
{
  "persistent": {
    "indices.lifecycle.poll_interval": "5s"
  }
}
Luc E
  • 1,204
  • 8
  • 16
  • In a scenario where limit has reached and the index became read only, then i delete all documents from the original index. How to make the rollover in the oposite direction? How to make it go back to being read-only=false? – Montoya Apr 09 '20 at 09:25
  • PUT my-index-000002/_settings { "index.blocks.read_only": false } – Luc E Apr 09 '20 at 10:42
  • This will make the index immediately writeable. I want that the threshold will be checked again by the policy and become writeable if the threshold isn't met. For example - if i put max 5 docs, and insert 5 docs in, making it read only. After that i delete 1 doc making index size 4. I want the policy to notice that, making an automatic rollover to becoming writeable – Montoya Apr 09 '20 at 14:13