0

I'm having a lot of issues tuning Elasticsearch to give a high search query performance. These are my specs:

ES Setup: Version: 0.90.0, 2 nodes (m3.2xlarge aws intances) in cluster, 32GB RAM each, 50% allocated to ES_HEAP_SIZE, no swapping

Data: 75MM documents, 25 fields each

Queries made for benchmark: Multimatch query against 5 text fields

I've tried everything mentioned here and here

Upto a 30 requests/sec input query frequency, the response time stays less than 1s. Above that 30+ requests/sec, the performance plummets and response time increases to 50s. While this happens JVM Heap is stable (around 7-8 in bigdesk) and GC is also stable. However, the CPU rapidly increases and is at 800% (8-core) and load average is very high 16. The hot threads keep switching between search and scoring functions like BooleanScorer2.nextDoc, BooleanQuery.createWeight, DisjunctionSumScorer.advance, BufferedIndexInput.refill and such

Question: Could you help me find out why performance plummets after 30 req/sec and how to resolve this by changing the cluster configuration if possible.

Thanks in advance!

alpha_cod
  • 1,933
  • 5
  • 25
  • 43
  • Can you increase size of search queue and see what is going on – Vineeth Mohan Nov 05 '15 at 03:24
  • Thanks Vineeth, for the suggestion. I tried increasing the thread pool peak for search from default 16 to 40. It did not make a difference to the max reqs/sec that I could support with sub-second latencies. The CPU still maxed out at 30 reqs/sec and response times dropped after that. – alpha_cod Nov 05 '15 at 23:31

1 Answers1

0

I know you're seeing CPU-bound behavior, but are you seeing any I/O spikes around when you start having performance issues?

If you're storing your index on EBS volumes I wouldn't be surprised to start seeing I/O saturation with a test like yours. M3 instances have fast local (ephemeral) SSD volumes, and if you're tuning for responsiveness, you should make sure your index is stored locally.

https://www.elastic.co/blog/performance-considerations-elasticsearch-indexing

I realize this doesn't speak directly to the CPU issue, but anything you can do to make a single query more responsive (including filtering/etc...) will boost your throughput.

Peter Dixon-Moses
  • 3,169
  • 14
  • 18