Can I set active master in an Elasticsearch cluster manually?

Question

I know that it is possible to define more than one master for the ElasticSearch cluster, where only one acts as master and the others can step in if necessary. See also https://stackoverflow.com/a/15022820/2648551 .

What I don't understand is how I can determine which master is active and which could step in if necessary.

The following setting I currently have:

node-01: master (x) data(-)
node-02: master (-) data(x)
node-03: master (-) data(x)
node-04: master (-) data(x)
node-05: master (-) data(x)
node-06: master (-) data(x)

Now I want to determine that e.g. node-02 becomes additionally a master eligible. Can I rely on ES being so smart that it always takes the non-data node (node-01) as the active master, or could it be that node-02 ever acts as the active master if all nodes are present and there are no problems? Or is that something I just don't have to worry about?

I am currently using ElasticSearch 1.7 [sic!], but I am also interested in answers based on the latest versions.

Answer 1

A few laters and just for the context we "can" now decide which node becomes master, although not straight forward its possible.

Elasticsearch now has an method called voting_config_exclusions which can be used to move away from current master node e.g.

lets say you have 3 master-eligible nodes in your cluster

$ GET _cat/nodes?v

ip               node.role     master   name
192.168.0.10     cdfhilmrstw   -        node-10
192.168.0.20     cdfhilmrstw   *        node-20
192.168.0.30     cdfhilmrstw   -        node-30
192.168.0.99     il            -        node-99

and Elasticsearch has selected node-20 as active master, you can run following call to remove the active node from voting.

POST /_cluster/voting_config_exclusions?node_names=node_name

This will randomly select another master-eligible node as master (if you have more than one left) Keep doing this for the active nodes until you get the right one activated as master.

Note: this doesn't remove the node, only makes it in-active master / non-voting node and allows another node to become active as master.

Once done, make sure to run below command to remove exclusions and allow all eligible nodes to become master if and when the selected node goes down.

DELETE /_cluster/voting_config_exclusions

Thank You

Answer 2

In short, no, you can't decide which of the master eligible nodes will become a master, because master node is elected (it was in ES 1.7, it still is in ES 6.2).

No, you can't rely on Elasticsearch being so smart to always take the non-data node as the active master. In fact, as of now (6.2) they advice to have dedicated master nodes (i.e. those that do not perform any data operations):

To ensure that your master node is stable and not under pressure, it is a good idea in a bigger cluster to split the roles between dedicated master-eligible nodes and dedicated data nodes.

... It is important for the stability of the cluster that master-eligible nodes do as little work as possible.

(Note that they are talking about a "bigger cluster".)

I can only assume that this also holds for the earlier versions and the documentation just got reacher.

There is a problem with the configuration that you have posted. Although you have many nodes, loss of one (the master node, node-01) will make your cluster non-functional. To avoid this situation you may choose one of these options:

• use default strategy and make all nodes data nodes and master nodes;
• make a set of dedicated master-only nodes (at least 3 of them).

It would be nice to know the reason why the ES defaults are not good enough for you, because usually they are good enough.

However, if this is the case when you need a dedicated master node, make sure you have at least 3 and that discovery.zen.minimum_master_nodes is enough to avoid the "split brain" situation:

discovery.zen.minimum_master_nodes = (master_eligible_nodes / 2) + 1

Hope that helps!