0

We are facing an issue with our ES. Merging is not happening as quickly as ingestion/updates and as a result we have a huge number of deleted documents that take an additional 65% space. I have read that merging happens automatically and we can also force with a ES command but both doesn't seem working unless I stop item ingestion/updates. ES gives great performance for our aggregation queries on millions of items and so we are using it as our primary DB.

We switched from ES 2.X to ES 5.5 and still have this issue.

I have played with forcemerge, shard size, shard count & stopping ingestion. Only the last one worked.

Is there any way for us to reduce this deleted documents count without stopping item ingestion/updates?

yalkris
  • 2,596
  • 5
  • 31
  • 51
  • I would contact elastic search support for this one – twoleggedhorse Aug 03 '17 at 23:46
  • Can you show what you get from `curl -XGET localhost:9200/_cat/segments?v` + `curl -XGET localhost:9200/_cat/allocation?v` + `curl -XGET localhost:9200/_cat/indices?v` – Val Aug 04 '17 at 04:01

1 Answers1

1

On Elasticsearch 5.x there should be a mechanism to backpressure index rates, if merging falls behind. Wondering if this is the case or not. There is another setting you can play around with, that decides when a merge should be triggered based on the number of deletes. You can configure this as part of the merge policy, see

https://github.com/elastic/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/index/MergePolicyConfig.java

alr
  • 1,744
  • 1
  • 10
  • 11
  • Thanks for your response. I went through that config and no where it says anything about backpressure. Is it a combination of those settings that will create that backpressure? – yalkris Aug 04 '17 at 18:00
  • 1
    See the javadocs for the ConcurrentMergeScheduler, https://lucene.apache.org/core/6_6_0/core/org/apache/lucene/index/ConcurrentMergeScheduler.html – alr Aug 06 '17 at 18:49