How to remove a node from an Elasticsearch cluster at runtime without downtime

Question

Suppose I had five nodes in a cluster and I had to remove two nodes at run time. So how can it be done without affecting the indices?

I had a continuous stream of data coming at nearly 10 Gb/hour which is getting indexed continuously.

Would rebalancing be a help in this?

why is solr tagged in this question? I guess this is only elasticsearch related question. Please don't use wrong tags. This might mislead users to look into irrevlevant question. — Vamsi Krishna, Jun 26 '13 at 07:37

score 149 · Answer 1 · edited Jun 26 '19 at 10:10

149

You can decommission a node by telling the cluster to exclude it from allocation. (From the documentation here)

curl -XPUT localhost:9200/_cluster/settings -H 'Content-Type: application/json' -d '{
  "transient" :{
      "cluster.routing.allocation.exclude._ip" : "10.0.0.1"
   }
}';echo

This will cause Elasticsearch to allocate the shards on that node to the remaining nodes, without the state of the cluster changing to yellow or red (even if you have replication 0).

Once all the shards have been reallocated you can shutdown the node and do whatever you need to do there. Once you're done, include the node for allocation and Elasticsearch will rebalance the shards again.

edited Jun 26 '19 at 10:10

mR_fr0g

8,462
7
39
54

answered May 28 '14 at 07:15

towr

4,147
3
20
30

37

Also "cluster.routing.allocation.exclude._name" and "cluster.routing.allocation.exclude._id" can be used to decommission by node name and node id. Needed this in a situation when due to incorrect configuration (/etc/hosts & elasticsearch.yml) all nodes got the same (published) IP 127.0.1.1. – Alexey Tigarev Nov 19 '14 at 16:28
3

How do I check if it is OK to shut down the decommissioned node? – Mischa Arefiev May 14 '15 at 14:02
17

You could use `curl -XGET 'http://ES_SERVER:9200/_cluster/health?pretty'`; if there are no nodes relocating, then the excluded node must have been drained and safe to shut down. Another option is to check `curl -XGET 'http://ES_SERVER:9200/_nodes/NODE_NAME/stats/indices?pretty'` to see if there are no documents left on the node. – towr May 14 '15 at 20:21
14

Multiple _ip records can be comma separated, ex: "cluster.routing.allocation.exclude._ip" : "10.0.0.1,10.0.0.2" – Marc Tamsky Oct 30 '15 at 20:54
If you have a cluster behind a load balancer, should you remove the node from the load balancer then remove it from routing allocation, or remove it from routing allocation then the load balancer? – Tom Miller Mar 12 '16 at 20:05
1

@TomMiller - It doesn't matter. if you take it out of the routing allocation before removing it from the load balancer it'll still handle queries and return results. – Paul Lemke Dec 22 '16 at 16:10
1

Note that starting from ES6, you need to also specify the content type, e.g. curl -XPUT localhost:9200/_cluster/settings -H "Content-Type: application/json" -d '{ "transient" :{ "cluster.routing.allocation.exclude._ip" : "10.0.0.1" } }';echo – dfsg76 Feb 02 '18 at 10:11
@pudelwudel Updated the answer to use appropriate `Content-Type` header. – Imran Feb 05 '18 at 14:53

score 11 · Answer 2 · edited Dec 29 '22 at 01:11

To remove an Elasticsearch node from the cluster. Just run the following command:

curl -XPUT P.P.P.P:9200/_cluster/settings -H 'Content-Type: application/json' -d '{
  "transient" :{
      "cluster.routing.allocation.exclude._ip" : "X.X.X.X"
   }
}';echo

Here P.P.P.P is the private IP address of the master node. You may also use the localhost if Elasticsearch is running on localhost. X.X.X.X is the private IP address of the node to be removed from the cluster.

This command will give acknowledgement true if the node is accepted to be removed and the data relocation will start. Check if the data relocation is over and the node doesn't have any shards left on it. Then stop the elasticsearch process and stop/terminate the instance.

The commands to check data relocation and shards left can be found in this article.

I do not think this is accurate. At least on v1.7.5 if you exclude a node it will not get NEW shards allocated to it but existing shards are not effected. — TheFiddlerWins, Nov 15 '16 at 15:31

How to remove a node from an Elasticsearch cluster at runtime without downtime

2 Answers2

Linked

Related