Elasticsearch multi node cluster using kubernetes?

Question

I am trying to create a multiple node cluster of elasticsearch. So which service should i use to create a cluster using kubernetes. I was able to do it within a node using headless service for the internal communication between es. But the same is not happening in the case of multiple node. Also which ip and port i have to mention in "discovery.zen.ping.unicast.hosts" of the master node in the elasticsearch.yml file in worker node.

deployment.yml file

apiVersion: apps/v1
kind: Deployment
metadata:
name: elasticsearch-deployment
spec:
 selector:
 matchLabels:
   app: elasticsearch
 replicas: 2
 template:
   metadata:
     labels:
       app: elasticsearch
   spec:
     containers:
     - name: elasticsearch
       image: sandeepp163/elasticsearch:latest
       volumeMounts:
       - mountPath: /usr/share/elasticsearch/config/
         name: config
       - mountPath: /var/logs/elasticsearch/
         name: logs
     volumes:
     - name: config
       hostPath:
         path: "/etc/elasticsearch/"
     - name: logs
       hostPath:
         path: "/var/logs/elasticsearch"

internal-communication service config

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-cluster
spec:
  clusterIP: None
  selector:
    app: elasticsearch
  ports:
  - name: transport
    port: 9300
    targetPort: 9300

external service config

apiVersion: v1
kind: Service
metadata:
  name: load-service
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  type: NodePort
  ports:
  - nodePort: 31234
    port: 9200
    targetPort: 9200

The error im getting on the worker node.

[2020-02-26T05:29:02,297][WARN ][o.e.d.z.ZenDiscovery     ] [worker] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again

elasticsearch.yml file in worker

cluster.name: xxx
node.name: worker
node.master: false
node.data: true
node.ingest: false
discovery.zen.ping.unicast.hosts: ["192.168.9.0"]
discovery.zen.minimum_master_nodes: 1

elasticsearch.yml in master

cluster.name: xxx
node.name: master
node.master: true
node.data: false
node.ingest: false

Thanks

Answer 1

You could simply deploy Elasticsearch in a statefulset using HELM.

1. Installing HELM:

If you are using linux, type: curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

If not, see here the installation process for your O.S.

2. Add the stable repository to Helm, and upgrade them:

helm repo add stable https://kubernetes-charts.storage.googleapis.com

helm repo update 

3. Install Elasticsearch Helm chart

Now you're able to install the Elasticsearch chart, type:

helm install stable/elasticsearch --generate-name

Wait for the installation, you could check using kubectl get pods -l app=elasticsearch

To access you could use proxy-port on service name:

ES_SVC=$(kubectl get svc -owide -l "app=elasticsearch" -o jsonpath="{.items[0].metadata.name}")

kubectl port-forward svc/$ES_SVC 9200:9200

4. Access the service:

To access the service go to http://127.0.0.1:9200 from your browser.

Hope that helps.