1

This is the error message I am facing while executing the Reindex call

[2020-03-13T16:08:39,042][WARN ][o.e.i.r.TransportReindexAction] [ExportDatabase] giving up on search because it failed with a non-retryable exception org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:288) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:128) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:249) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:48) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:222) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:176) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:48) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:222) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:527) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1095) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1188) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1172) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:66) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.SearchTransportService$6$1.onFailure(SearchTransportService.java:385) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:341) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:329) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService$3.doRun(SearchService.java:1019) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.0.jar:6.3.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_212] at java.lang.Thread.run(Unknown Source) [?:1.8.0_212] Caused by: org.elasticsearch.ElasticsearchException$1: Determinizing automaton with 33501 states and 34977 transitions would result in more than 10000 states. at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:658) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:126) ~[elasticsearch-6.3.0.jar:6.3.0] ... 26 more Caused by: org.apache.lucene.util.automaton.TooComplexToDeterminizeException: Determinizing automaton with 33501 states and 34977 transitions would result in more than 10000 states. at org.apache.lucene.util.automaton.Operations.determinize(Operations.java:746) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24] at org.apache.lucene.util.automaton.RunAutomaton.(RunAutomaton.java:69) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24] at org.apache.lucene.util.automaton.CharacterRunAutomaton.(CharacterRunAutomaton.java:39) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24] at org.apache.lucene.util.automaton.CharacterRunAutomaton.(CharacterRunAutomaton.java:28) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24] at org.elasticsearch.common.xcontent.support.XContentMapValues.filter(XContentMapValues.java:174) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.fetch.subphase.FetchSourceContext.getFilter(FetchSourceContext.java:234) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.lookup.SourceLookup.filter(SourceLookup.java:132) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.fetch.subphase.FetchSourceSubPhase.hitExecute(FetchSourceSubPhase.java:56) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:162) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:393) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:368) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:333) ~[elasticsearch-6.3.0.jar:6.3.0] ... 9 more

So as per my analysis this is the query which I am executing:

POST _reindex
{
"source":{
"index":"index_source",
"size":1000,
"query":
{
"terms":{
"ID":["7078"],
"boost":1.0
}
},
"_source":
{
"includes":here we have 1216 fields]
}
},
"dest":
{
"index":"index_dest"
}
}

Here if I remove the 1216 fields in the include param or make it less to around 500 then it works

So need help in this, is there any other way we can only reindex for these specific fields or is there something wrong I am doing.

HRS
  • 23
  • 4

1 Answers1

0

From elastic search documentation Mapping

1000 is maximum number of fields allowed in an index. To increase the limit you can update "index.mapping.total_fields.limit"

Warning

The limit is in place to prevent mappings and searches from becoming too large. Higher values can lead to performance degradations and memory issues, especially in clusters with a high load or few resources. If you increase this setting, we recommend you also increase the indices.query.bool.max_clause_count setting, which limits the maximum number of boolean clauses in a query.

jaspreet chahal
  • 8,817
  • 2
  • 11
  • 29
  • Hello @jaspreetchahal, thanks for your prompt reply. But I already increased the field limit and checked it does not work. My current field limit is 4000. Also I observed that with 960 terms, it works. – HRS Mar 18 '20 at 09:52
  • Currently the field limit is as below: Source Index: "index": { "mapping": { "total_fields": { "limit": "4636" } } Destination index: "index": { "mapping": { "total_fields": { "limit": "4000" } } But I indexing only 1216 terms in the reindex call – HRS Mar 18 '20 at 09:58
  • Is this field limit in new or old index? – jaspreet chahal Mar 18 '20 at 10:30
  • From reindex docs :-"You must set up the destination index before calling _reindex. Reindex does not copy the settings from the source index. Mappings, shard counts, replicas, and so on must be configured ahead of time." While reindexing settings and mappings are not copied. If mapping for destination index doesn't exist then mapping is created based on first document copied so you will loose formatting type of date etc. So you should create destination index first before copying the data – jaspreet chahal Mar 18 '20 at 10:46
  • 4000 is in the new index and the 4636 is in the old index. Yes, I have created the destination index already with all the respective mapping. Also in the reindex call if I keep **"includes":empty** then also it works but I get all the attributes from the source index which I don't want. – HRS Mar 18 '20 at 11:01
  • I tried with "includes": "" no field got copied. removing includes or passing "includes": "*" copies all the field. Since you mentioned you are getting all attributes from source index. is your original issue resolved? – jaspreet chahal Mar 18 '20 at 11:17
  • Please try "includes":[] then all your fields will get copied. No I don't want all the fields I want only those 1216 fields, so still my issue persists. I want to know if there is any limit from the Elastic side for this include parameter or is it configurable. – HRS Mar 18 '20 at 12:04
  • I did a little research on this. source values are considered to be patterns so there is limit of max_determined_states. In some query (wildcard etc) flexibility is given to increase this limit but this option is not present in source. I have opened a bug ticket in github for this https://github.com/elastic/elasticsearch/issues/53739. – jaspreet chahal Mar 20 '20 at 09:16
  • Creating 1000 fields in index can lead to mapping explosion and can seriously impact performance. I will suggest use of client application to index (in place of reindex) and reducing the number of fields. Link for further study on mapping explosion https://www.elastic.co/blog/found-crash-elasticsearch – jaspreet chahal Mar 20 '20 at 09:17
  • Thanks @jaspreetchahal I am also working on the same thing, currently reduced the no. of terms. But this not a solution just a workaround. I will go through elastic.co/blog/found-crash-elasticsearch – HRS Mar 23 '20 at 14:04
  • Ideally there should not be so many fields as elasticsearch relies on memory it will create problem. You should try to nest documents(object or nested field) or create parent child relationship or split it multiple index – jaspreet chahal Mar 23 '20 at 14:11
  • Ex if there are multiple fields ex answer1, answer2,answer3.... these can be a single field answer:["value1","value2"] – jaspreet chahal Mar 23 '20 at 14:19
  • The default value is not 1000, but 10000, you may check this from elastic Official document: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-settings-limit.html – Timothy.Li Jun 08 '21 at 06:22
  • @Timothy.Li index.mapping.total_fields.limit = 1000 , index.mapping.nested_objects.limit = 10000 as per document – jaspreet chahal Jun 14 '21 at 07:11