0

I'm still relatively new to Elasticsearch and, currently, I'm attempting to switch from Solr to Elasticsearch and am seeing a huge increase in CPU usage when ES is on our production website. The site sees anywhere from 10,000 to 30,000 requests to ES per second. Solr handles that load just fine with our current hardware.

The books index mapping: https://pastebin.com/bKM9egPS
A query for a book: https://pastebin.com/AdfZ895X

ES is hosted on AWS on an m4.xlarge.elasticsearch instance.
Our cluster is set up as follows (anything not included is default): 

"persistent": {
"cluster": {
  "routing": {
    "allocation": {
      "cluster_concurrent_rebalance": "2",
      "node_concurrent_recoveries": "2",
      "disk": {
        "watermark": {
          "low": "15.0gb",
          "flood_stage": "5.0gb",
          "high": "10.0gb"
        }
      },
      "node_initial_primaries_recoveries": "4"
    }
  }
},
"indices": {
  "recovery": {
    "max_bytes_per_sec": "60mb"
  }
}

Our nodes have the following configuration:

  "_nodes": {
    "total": 2,
    "successful": 2,
    "failed": 0
  },
  "cluster_name": "cluster",
  "nodes": {
    "####": {
      "name": "node1",
      "version": "6.3.1",
      "build_flavor": "oss",
      "build_type": "zip",
      "build_hash": "####",
      "roles": [
        "master",
        "data",
        "ingest"
      ]
    },
    "###": {
      "name": "node2",
      "version": "6.3.1",
      "build_flavor": "oss",
      "build_type": "zip",
      "build_hash": "###",
      "roles": [
        "master",
        "data",
        "ingest"
      ]
    }
  }

Can someone please help me figure out what exactly is happening so I can get this deployment finished?

Nathan
  • 2,699
  • 5
  • 30
  • 44
  • 1) 10K to 30K search requests/ sec ? 2) Do you have time based indices ? 3) What is current indexing rate ? 4) Do you encounter any search thread pool exhaustion errors when invoking search queries on your cluster ? 5) How many shards do you have per node ? 6) What is avg shard size ? Please note each shard in itself consumes cluster resources such as cpu etc. Elastic recommends shard size between 10 - 40 GB for optimal performance not exceeding 50 GB – ben5556 Dec 14 '18 at 08:06
  • @ben5556 1) Yes. 2) No. 3) Indexing isn't something that happens on the regular, but the rate when building the indices peaked at 29.5k and averages around 15k. 4) No. 5) 1 shard per index with 1 replica on 5 indices so I think that means 10 shards per node? (I'm not quite sure) 6) Shard size varies greatly with the smallest being 1MB and largest being 5GB. – Nathan Dec 14 '18 at 17:32

0 Answers0