1

I have K8s cluster with 3 nodes "VMs" doing both master/worker with Etcd installed on all of the 3 nodes "untainted master" installed via kubespray "kubeadm based tool"

Now I would like to replace one VM with another. Is there a direct way to do so because the only workaround will be adding 2 nodes to have always an odd number of ETCD via kubespray scale.yml ex:node4 and node 5 then removing the additional node3 and node 5 keeping node 4

I don't like the approach.

Any ideas are welcomed

Best Regards

Abdennour TOUMI
  • 87,526
  • 38
  • 249
  • 254
Swiftree
  • 11
  • 1
  • I'm not too familiar with kubespray so I hestitate to answer definitely, but you should be fine temporarily having an even number of nodes. Its just that an even number is not more highly available than an odd number (because quorum is >50%), so having an even number is sort of wasteful from an availability standpoint. Since you're talking about temporarily going up to 4 and then back down to 3, it should be fine. – Brian Pursley Jul 15 '20 at 20:02

1 Answers1

2

If you have 3 main (please avoid using master, ) control plane nodes you should be fine replacing 1 at a time. The only thing is that your cluster will not be able to make any decisions/schedule any new workload, but the existing workloads will run fine.

The recommendation of 5 main nodes is based on the fact that you will always have a majority to reach quorum on the state decisions for etcd even if one node goes down. So if you have 5 nodes and one of them goes down you will still be able to schedule/run workloads.

In other words:

  • 3 main nodes

    • Can tolerate a failure of one node.
    • Will not be able to make decisions

  • 5 main nodes

    • Can tolerate a failure of one node.
    • Can still make decisions because there are 4 nodes still available.
    • If 2 failures happen then it's tolerated but there is no quorum.

To summarize, Raft which is the consensus protocol for etcd tolerates up to (N-1)/2 failures and requires a majority or quorum of (N/2)+1. The recommended procedure is to update the nodes one at a time: bring one node down, and then a bring another one up, and wait for it to join the cluster (all control plane components)

Rico
  • 58,485
  • 12
  • 111
  • 141
  • Thanks Rico for your answer but what is the procedure for doing the replacement – Swiftree Jul 15 '20 at 20:22
  • Added at the bottom one at a time. Using your kubespray approach. I believe it's too much hassle to just another 2 nodes because you can always schedule a time when you are not deploying anything to a cluster. The thing that you typically care the most is uptime of your services running in the cluster – Rico Jul 15 '20 at 20:30