1

I have set up elastic search using official Elastic Search Helm chart I followed below steps

  1. Created cluster as below

    gcloud container clusters create cluster-1 --enable-autoscaling --zone us-central1-c \ --num-nodes 2 \ --min-nodes 1 \ --max-nodes 5

  2. used custom values.yaml for installing elastic search.

helm install elasticsearch elastic/elasticsearch -f ./values.yaml

values.yaml

replicas: 1
# Shrink default JVM heap.
esJavaOpts: "-Xmx128m -Xms128m"

# Allocate smaller chunks of memory per pod.
resources:
  requests:
    cpu: "100m"
    memory: "512M"
  limits:
    cpu: "200m"
    memory: "512M"

# Request smaller persistent volumes.
volumeClaimTemplate:
  accessModes: [ "ReadWriteOnce" ]
  storageClassName: "standard"
  resources:
    requests:
      storage: 500M

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  path: /
  hosts:
    - elasticsearch.mydomain.com

  1. Created index and inserted document it elastic db.

    curl -X PUT "http://elasticsearch.mydomain.com/company" -H 'Content-Type: application/json' -d' { "settings": { "number_of_shards": 1, "number_of_replicas": 1 }, "mappings": { "properties": { "age": { "type": "integer"}, "experienceInYears": {"type": "integer" }, "name": {"type": "text"} } } } '

    curl -X PUT "http://elasticsearch.mydomain.com/company/_doc/1" -H 'Content-Type: application/json' -d' { "name": "John", "age" : 30, "experienceInYears" : 6 } '

  2. Now I want to send bulk search request to generate load on elastic search pod. So I written below shell script and tried to execute from multiple terminal.

`

#!/bin/bash
i=0 # set counter to 0
while true # infinite loop
do   
    curl -X GET "http://elasticsearch.mydomain.com/company/_search?pretty" -H 'Content-Type: application/json' -d'
    {
        "query" : {
            "match" : { "name" : "John" }
        }
    }' 
    if [ $? -ne 0 ]
    then
        # curl didn't return 0 - failure
        echo $i
        break # terminate loop
    fi
    i=$(($i+1))  # increment counter
    echo -en "$i        \r"   # display # of requests each iteration
    sleep 0.25  # short pause between requests
done

`

  1. Now as I am sending bulk request new node should get created (kubernetes cluster auto scaling), but its showing only 2 nodes. May be because cpu limit of 200m is not crossed.

     kubectl get nodes
 NAME                                       STATUS   ROLES    AGE   VERSION
 gke-cluster-1-default-pool-d6891b13-flf9   Ready    <none>   18h   v1.14.10-gke.36
 gke-cluster-1-default-pool-d6891b13-hw2x   Ready    <none>   18h   v1.14.10-gke.36

 kubectl top pods
 NAME                                                              CPU(cores)   MEMORY(bytes)
 elasticsearch-master-0                                            160m          371Mi
 ingress-elastic-search-test-ingress-nginx-controller-5c96ctthr2   25m           68Mi

So how to auto scale elastic search cluster based on load. How to generate load on elastic search pods ?

mario
  • 9,858
  • 1
  • 26
  • 42
Developer Desk
  • 2,294
  • 8
  • 36
  • 77
  • Does this answer your question? [How to autoscale elastic search in kubernates based on load?](https://stackoverflow.com/questions/62710785/how-to-autoscale-elastic-search-in-kubernates-based-on-load) – David Maze Jul 22 '20 at 11:38
  • You need to cause Elasticsearch to create more replicas. Scaling _up_ isn't necessarily tricky but scaling _down_ is hard and requires database-specific knowledge, and can take a long time relative to changes in load. – David Maze Jul 22 '20 at 11:39
  • OK cluster auto scaling is not possible in elastic search ? You mean to say i should use replicas instead of cluster auto scaling ? – Developer Desk Jul 22 '20 at 11:45
  • _Cluster_ autoscaling causes the cluster to create more nodes if pods' resource requests require them. You need _pod_ autoscaling here, and that's tricky for Elasticsearch (I detail this in the linked question). – David Maze Jul 22 '20 at 11:47

0 Answers0