1

I've a decently sized ES index (10TB) with 50 split on 50 machines (1 shard each), close to 10B rows. Machines are top-tier (the largest you can get on AWS). RAM per ES instance is set to 30 GB.

Whenever I run a very simple query such as :

POST /my_index/_search
{
  "size": 1, 
  "query": {
    "match_all": {}
  }
}

It takes between 2 to 20+ seconds (I even got 502):

Response

{
  "took" : 17584,
  "timed_out" : false,
  "_shards" : {
    "total" : 50,
    "successful" : 50,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : 1.0,
    "hits" : [
      {

Is there a way to make it faster? I noticed that the LIMIT clause is not working great in ElasticSearch (or better, I'm probably not using it right)

M4rk
  • 2,172
  • 5
  • 36
  • 70
  • 2
    Using `LIMIT` without an `ORDER BY` is silly. Anyway, it's slow because it's scanning all shards. – Dai Oct 22 '20 at 01:43
  • @Dai exactly, I also mentioned this in my answer :) pls hv a look – Amit Oct 22 '20 at 01:45
  • @Dai Are you suggesting that an order by would make it quick? This is a sample query, meant to be silly, but there can be similar cases, like in paging. I don't believe Postgres would need to scan all DBs for this and it is always fast. – M4rk Oct 22 '20 at 04:07
  • @rodi What is this query meant to accomplish though? Are you just wanting to see if the ES index _contains anything_ - or something else? – Dai Oct 22 '20 at 04:48
  • @Dai The goal is: the user might make very generic queries that would match hundreds of millions of results before making queries that will match only a subset. I would like that those queries didn't take long. E.g. You look for "elasticsearch" on google, and then you look "elasticsearch error12345". – M4rk Oct 22 '20 at 13:18

1 Answers1

1

Interesting ES performance question, please clarify a few things, and based on my understanding I will try to explain the things

  1. When you mentioned RAM per ES instance is set to 30 GB, I guess you meant ES heap size is 30 GB, not the ES node's RAM size? as thumb rule is to assign 50% RAM of the node to ES Heap size and it(ES heap) shouldn't cross 32 GB.
  2. Hope you have tried this query during peak and non-peak hours and your range is including both the time-frame?

Now, few recommendations to speed up your search query

  1. Try to reduce the number of shards for your index, currently its 50 shards, this means the coordinating node has to collect the search result from 50 nodes(your case of 1 shard each node)and this inter-node communication might be taking quite some time.

  2. Have written short tips to improve search performance, and see if you can apply to your cluster, index

Amit
  • 30,756
  • 6
  • 57
  • 88
  • 1) Yes, heap. RAM is a couple fold higher 2) dev env 3) If I halven it, the storage per node will be higher than 40GB, would that be a good idea? And the usage is subject to grow 4) Thank you! – M4rk Oct 22 '20 at 04:14
  • @rodi reg 3) its disk storge which will grow and that's fine, please do it, measure it and let us know the gain :) – Amit Oct 22 '20 at 04:31
  • Ok, requirements now asked to put 10TB of data in it. reducing the number of nodes would result in >400GB per node. From Elastic guidelines it seems we might even have to add more nodes: https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html#shard-size-recommendation – M4rk Oct 22 '20 at 15:37
  • @rodi, i don't understand why you want to keep 10 TB of data in just single index, ideally you should create a new index with few shards, so that you have to search fewer shards, please note my recommendation is to reduce **shards not nodes** – Amit Oct 22 '20 at 16:50
  • We need because the data is all of the same kind (same mapping), so I would imagine it should stay in the same index; breaking it in multiple indexes would mean partitioning the data – M4rk Oct 22 '20 at 18:48
  • @rodi could you please provide update on my last comment – Amit Oct 26 '20 at 06:19