Elasticsearch high cpu usage during load test (50 req/sec)

Question

I have some trouble with elasticsearch.

Currently:

I have a machine specs as follows

CPU vendor: Intel
CPU model: Xeon (2001 MHz)
CPU total logical cores: 24
CPU cache: 15kb
VM name: Java HotSpot(TM) 64-Bit Server VM
VM vendor: Oracle Corporation
VM version: 25.31-b07
Java version: 1.8.0_31

elasticsearch v/1.3.2

I have index(metadatav3) with:

settings: http://pastebin.com/6Q9f3tPv

mapping

{
  "metadatav3": {
    "mappings": {
      "track": {
        "dynamic": "true",
        "numeric_detection": true,
        "properties": {
          "album.album": {
            "type": "string",
            "norms": {
              "enabled": false
            },
            "analyzer": "music_field"
          },
          "album.exact": {
            "type": "string",
            "analyzer": "exact_music_field"
          },
          "artist.artist": {
            "type": "string",
            "norms": {
              "enabled": false
            },
            "analyzer": "music_field"
          },
          "artist.exact": {
            "type": "string",
            "analyzer": "exact_music_field"
          },
          "fullString": {
            "type": "string",
            "norms": {
              "enabled": false
            },
            "analyzer": "nGram_token_field"
          },
          "fullString.token": {
            "type": "string",
            "norms": {
              "enabled": false
            },
            "analyzer": "music_field"
          },
          "id": {
            "type": "string"
          },
          "isHidden": {
            "type": "boolean"
          },
          "lastRankedDate": {
            "type": "long"
          },
          "popularity": {
            "type": "float"
          },
          "tagCount": {
            "type": "long"
          },
          "title.edgeNGNoSplit": {
            "type": "string",
            "norms": {
              "enabled": false
            },
            "analyzer": "edge_nGram_no_split_small_field"
          },
          "title.exact": {
            "type": "string",
            "analyzer": "exact_music_field"
          },
          "title.title": {
            "type": "string",
            "norms": {
              "enabled": false
            },
            "analyzer": "music_field"
          }
        }
      }
    }
  }
}

When I run that query:

{
  "from": 0,
  "size": 20,
  "timeout": 5000,
  "query": {
    "function_score": {
      "query": {
        "bool": {
          "must": {
            "match": {
              "fullString": {
                "query": "test",
                "type": "boolean",
                "operator": "OR",
                "minimum_should_match": "1",
                "cutoff_frequency": 0.01
              }
            }
          },
          "must_not": {
            "term": {
              "isHidden": "true"
            }
          },
          "should": []
        }
      },
      "field_value_factor": {
        "field": "popularity"
      }
    }
  },
  "explain": false
}

near 50-60 req/sec search response times become 60ms to 4-5 secs.

However, when I run that query:

{
  "from": 0,
  "size": 20,
  "timeout": 5000,
  "query": {
    "function_score": {
      "query": {
        "bool": {
          "must": {
            "match": {
              "fullString.token": {
                "query": "test",
                "type": "boolean",
                "operator": "OR",
                "minimum_should_match": "1",
                "cutoff_frequency": 0.01
              }
            }
          },
          "must_not": {
            "term": {
              "isHidden": "true"
            }
          },
          "should": []
        }
      },
      "field_value_factor": {
        "field": "popularity"
      }
    }
  },
  "explain": false
}

I can see 600 req/sec during load test.

I mean the part that I don't understand that does using ngram filter can create that much cpu usage?

Also hot thread dumps are as follows: http://pastebin.com/5sFEZJa5 and I can also send/upload bigdesk screenshots.

Thanks.

looks like `ngram` is being applied to the search term too which means the number of terms being queried in the first is not the same as the second so comparison is not fair and the behavior is probably not what you want. Try setting the `analyzer` field in the first `match` query example `"match":{"fullString":{"query":"test","type":"boolean","operator":"OR","minimum_should_match":"1","cutoff_frequency":0.01,"analyzer" : "music_field"}` and see if it makes a difference — keety, Oct 22 '15 at 04:01
but the cpu load change is dramatic. I mean token count is increasing but not that much, right? — mehmetgunturkun, Oct 22 '15 at 06:52
it would depend on how many more hits that would result in right ? if the three token increase with an `OR` results in a substantially larger set of hit it would mean substantially more to sort the score . how many hits does the first query have vs the second query ? — keety, Oct 22 '15 at 13:50
yeah you are right when I query with "rihanna" I got 2172 hits from first one and 452987 from second one. But I mean if I query one it lasts 40ms. If ı set threadpool to "fixed" instead of "cached" doesn't that should fix the cpu usage problem? I mean query will take more time under load because it should wait in queue but elasticsearch cpu usage should be ok or I am missing something big? — mehmetgunturkun, Oct 22 '15 at 16:09
to me it kind of makes sense the cpu increase. Think about it as you increase the number of requests that are cpu intensive thre would be more context switching between the busy threads servicing the request. The increase in response could be attributed to the wait in queue and context switching. However to be honest,would recommend posting your question in [elasticsearch forum](https://discuss.elastic.co/c/elasticsearch). The es developers would have a better idea and provide a better answer than my speculation. If you do get a good response please do update the answer here :) — keety, Oct 22 '15 at 17:11
I have already created a thread over there: https://discuss.elastic.co/t/many-slow-queries-and-high-cpu-load-on-load-test-50req-sec/32624/3 Nobody have replied so far. — mehmetgunturkun, Oct 22 '15 at 17:16
aah i see. How many nodes are there in cluster ? Is there adequeate memory in the box ? Is swap being used ? Are you doing a [search_type](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-search-type.html#search-request-search-type) query_then_fetch or query_and_fetch . Probably you can try the latter and see if it affects the cpu usage. — keety, Oct 22 '15 at 17:42
Also how may shards do you have for the index. It looks like one thread is used per shard . Probably reducing the shard could reduce cpu-usage. Note i'm not suggesting more shards is bad it is just to help debug where the cpu usage may be coming from for the query. — keety, Oct 22 '15 at 17:42
Currently we are on beta environment. This is why, we only have one node. but we are going to have two nodes on prod. There are 5 shards but we wanted to keep shard count 5 in case we can put additional nodes in future. Actually that was another thing confuses me, one node and five shard could create that much cpu load if it has a possibility I can reduce the shard count and try after that. For debugging I can send a screenshot of bigdesk or HQ output during the load test or a thread dump. — mehmetgunturkun, Oct 22 '15 at 18:05
yeah would be interesting to see in one node how one shard performs under load. I would guess the cpu usage might decrease but not response time. Index time should be really slow. — keety, Oct 22 '15 at 18:17
first thing I will reindex with one shard, also while using one shard, I guess it does not mean to create replicas, right? — mehmetgunturkun, Oct 22 '15 at 21:05
hey keezy, I tested same index with shard = 1 and replica = 0, rps got even less and response times almost same, still investigating, just for your information. — mehmetgunturkun, Oct 27 '15 at 12:11
interesting rps got even less does that mean cpu usage increased ? — keety, Oct 27 '15 at 13:33

Elasticsearch high cpu usage during load test (50 req/sec)

mapping

0 Answers0