3

I am trying to index json documents in ElasticSearch with dynamic mappings on. Some of the documents have unpredictable number of keys (nested levels) because of which I started getting this error from ES Java api.

[ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] in index [my_index] has been exceeded]]]failure in bulk execution

I was wondering if there is an option which can be configured at index level where I can define to scan for fields only till certain level (maybe 2) and store the rest of the document as a string or in flattened form. I did come across some settings like index.mapping.depth.limit but it seems if I set it to 2, this setting rejects the document if there are more levels. link

pratpor
  • 1,954
  • 1
  • 27
  • 46

2 Answers2

3

For Total Field 

PUT <index_name>/_settings
{
  "index.mapping.total_fields.limit": 2000
}

For depth limit

PUT <index_name>/_settings
{
  "index.mapping.depth.limit": 2
}

https://www.elastic.co/guide/en/elasticsearch/reference/master/mapping.html

dhamo
  • 545
  • 2
  • 7
  • Came across this as mentioned in the question. This document rejects the documents for indexing if the depth is found more than 2 then. Here is the link https://discuss.elastic.co/t/index-mapping-depth/145766 – pratpor Dec 26 '19 at 05:55
  • You can make it allow inserting with the mapping setting `"dynamic": true` this will allow it to save all information, but index only a certain depth, this only this depth will be searchable. – lauksas Apr 20 '20 at 18:10
  • This is to increase the limit after the index has been created. Is there a way to change the default setting so that new indices automatically pick the updated value? – Rahul K Pandey Feb 09 '22 at 10:30
  • To ensure that index settings are set for each new index you have to create an `_index_template` - or `_template (legacy)` . See https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html – sastorsl Oct 07 '22 at 07:55
1

Add this to your index _settings:

"settings": {
    "index.mapping.nested_fields.limit": 150,
    ...
}
"mappings": {
...
}
Alex
  • 11
  • 2