ElasticSearch cluster move to new node type

Question

I have a cluster of 8 nodes in ec2. 4 per AZ with cluster.routing.allocation.awareness.attributes: aws_availability_zone I want to migrate all the shards to a different node type. So I set up to new nodes and added them to the cluster. Some shards are now moving to the new nodes. The end goal is to shut down all the old nodes with all the shards being split between two nodes. What's the best way to do that without shutting down nodes one by one and placing shards on nodes to be shut down soon?

EDIT: I suppose "cluster.routing.allocation.exclude._ip": "x.x.x.x, y.y.y.y, z.z.z.z" should work for me?

Answer 1

so cluster.routing.allocation.exclude._ip might work fine but you might have to force a reshuffle but honestly the best way to do it once the new nodes are online is just kill a single old node. The cluster will go into a yellow state but it will still work

If you have a setup of active shard with a single backup shard. Yellow just means it can't map all the backups to active or a backup doesn't have an active. So it re-does them and soon the cluster will go green again.

I've upgraded live production clusters this way

Have 3 m3.large cluster members for example. Bring up 3 new m4.2xlarge members. Kill a single m3.large and wait till the cluster is green. Then repeat for the last two m3.large. I've never had a problem

Answer 2

The solution is to

curl -XPUT localhost:9200/_cluster/settings -d '{
    "transient" : {
        "cluster.routing.allocation.exclude._ip" : "172.31.3.53,172.31.32.44,172.31.32.43,172.31.41.155,172.31.41.152,172.31.15.219,172.31.6.166,172.31.15.72"
    }
}'

Got it from source code on Github:

96 clusterExcludeFilters = DiscoveryNodeFilters.buildFromKeyValue(OR, excludeMap);

and

String[] values = Strings.splitStringByCommaToArray(entry.getValue());