Number of nodes AWS Elasticsearch

Question

I read documentation, but unfortunately I still don't understand one thing. While creating AWS Elasticsearch domain, I need to choose "Number of nodes" in "Data nodes" section. If i specify 3 data nodes and 3-AZ, what it actually means? I have suggestions:

1. I'll get 3 nodes with their own storages (EBS). One of node is master and other 2 are replicas in different AZ. Just copy of master, not to lose data if master node become broken.

2. I'll get 3 nodes with their own storages (EBS). All of them will work independent and on their storadges are different data. So at the same time data can be processed by different nodes and store on different storages.

It looks like in other AZ's should be replicas. but then I don't understand why I have different values of free space on different nodes

Please, explain how it works. Many thanks for any info or links.

Answer 1

I haven't used AWS Elasticsearch, but I've used the Cloud Elasticsearch service.

When you use 3 AZ (availability zones), means that your cluster will use 3 zones in order to make it resilient. If one zone has problems, then the nodes in that zone will have problems as well.

As the description section mentions, you need to specify multiples of 3 if you choose 3 AZ. If you have 3 nodes, then every AZ will have one zone. If one zone has problems, then that node is out, the two remaining will have to pick up from there.

Now in order to answer your question. What do you get with these configurations. You can check so yourself. Use this via kibana or any HTTP client

GET _nodes

Check for the sections:

• nodes.roles
• nodes.attributes

In the various documentations, blog posts etc you will see that for production usage, 3 nodes and 3 AZ is a good starting point in order to have a resilient production cluster.

So let's take it step by step:

• You need an even number of master nodes in order to avoid the split brain problem.
• You need more than one node in your cluster in order to make it resilient (if the node is unavailable).

By combining these two you have the minimum requirement of 3 nodes (no mention of zones yet).

But having one master and two data nodes, will not cut it. You need to have 3 master-eligible nodes. So if you have one node that is out, the other two can still form a quorum and vote a new master, so your cluster will be operational with two nodes. But in order for this to work, you need to set your primary shards and replica shards in a way that any two of your nodes can hold your entire data.

Examples (for simplicity we have only one index):

1. 1 primary, 2 replicas. Every node holds one shard which is 100% of the data
2. 3 primaries, 1 replica. Every node will hold one primary and one replica (33% primary, 33% replica). Two nodes combined (which is the minimum to form a quorum as well) will hold all your data (and some more)

You can have more combinations but you get the idea.

As you can see, the shard configuration needs to go along with your number and type of nodes (master-eligible, data only etc).

Now, if you add the availability zones, you take care of the problem of one zone being problematic. If your cluster was as a whole in one zone (3 nodes in one node), then if that zone was problematic then your whole cluster is out.

If you set up one master node and two data nodes (which are not master eligible), having 3 AZ (or 3 nodes even) doesn't do much for resiliency, since if the master goes out, your cluster cannot elect a new one and it will be out until a master node is up again. Now for the same setup if a data node goes out, then if you have your shards configured in a way that there is redundancy (meaning that the two nodes remaining have all the data if combined), then it will work fine.

Answer 2

Your answers should be covered in following three points.

If i specify 3 data nodes and 3-AZ, what it actually means?

• This means that your data and replica's will be available in 3AZs with none of the replica in same AZ as the data node. Check this link. For example, When you say you want 2 data nodes in 2 AZ. DN1 will be saved in (let's say) AZ1 and it's replica will be stored in AZ2. DN2 will be in AZ2 and it's replica will be in AZ1.

It looks like in other AZ's should be replicas. but then I don't understand why I have different values of free space on different nodes

• It is because when you give your AWS Elasticsearch some amount of storage, the cluster divides the specified storage space in all data nodes. If you specify 100G of storage on the cluster with 2 data nodes, it divides the storage space equally on all data nodes i.e. two data nodes with 50G of available storage space on each.

• Sometime you will see more nodes than you specified on the cluster. It took me a while to understand this behaviour. The reason behind this is when you update these configs on AWS ES, it takes some time to stabilize the cluster. So if you see more data or master nodes as expected hold on for a while and wait for it to stabilize.

Answer 3

Thanks everyone for help. To understand how much space available/allocated, run next queries:

GET /_cat/allocation?v
GET /_cat/indices?v
GET /_cat/shards?v

So, if i create 3 nodes, than I create 3 different nodes with separated storages, they are not replicas. Some data is stored in one node, some data in another.