1

I've started learning kubernetes with docker and I've been thinking, what happens if master node dies/fails. I've already read the answers here. But it doesn't answer the remedy for it.

Who is responsible to bring it back? And how to bring it back? Can there be a backup master node to avoid this? If yes, how?

Basically, I'm asking a recommended way to handle master failure in kubernetes setup.

Abhishek
  • 6,912
  • 14
  • 59
  • 85
  • 1
    Have you checked the official docs? https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/ha-topology/ – BogdanL Aug 14 '20 at 11:02
  • Nice. So, there is a way to add load balancer for master nodes as well. Good to know, thank you for the doc link! – Abhishek Aug 14 '20 at 11:08
  • Yes, Kubernetes is designed with high availability in mind, but in the end it's your responsibility to properly install and configure it. – BogdanL Aug 14 '20 at 11:11

3 Answers3

5

You should have multiple VMs serving as master node to avoid single point of failure.An odd number of 3 or 5 master nodes are recommended for quorum. Have a load balancer in-front of all the VMs serving as master node which can do load balancing and in case one master node dies loadbalancer should remove the VMs IP and make it as unhealthy and not send traffic to it.

Also ETCD cluster is the brain of a kubernetes cluster. So you should have multiple VMs serving as ETCD nodes. Those VMs can be same VMs as of master node or for reduced blast radius you can have separate VMs for ETCD. Again the odd number of VMs should should be 3 or 5. Make sure to take periodic backup of ETCD nodes data so that you can restore the cluster state to pervious state in case of a disaster.

Check the official doc on how to install a HA kubernetes cluster using Kubeadm.

HA with stacked ETCD enter image description here

Arghya Sadhu
  • 41,002
  • 9
  • 78
  • 107
1

In short, for Kubernetes you should keep master nodes to function properly all the time. There are different methods to make copies of master node, so it is available on failure. As example check this - https://kubernetes.io/docs/tasks/administer-cluster/highly-available-master/

Satyajit Das
  • 2,740
  • 5
  • 16
  • 30
GintsGints
  • 807
  • 7
  • 15
1

Abhishek, you can run master node in high availability, you should set up the control plane aka master node behind Load balancer as first step. If you have plans to upgrade a single control-plane kubeadm cluster to high availability you should specify the --control-plane-endpoint to set the shared endpoint for all control-plane nodes. Such an endpoint can be either a DNS name or an IP address of a load-balancer.

By default because of security reasons the master node does not host PODS and if you want to enable hosting PODS on master node you can run the following command to do so.

kubectl taint nodes --all node-role.kubernetes.io/master

If you want to manually restore the master make sure you back up the etcd directory /var/lib/etcd. You can restore this on the new master and it should work. Read about high availability kubernetes over here.

Abhishek
  • 6,912
  • 14
  • 59
  • 85
A Modgil
  • 260
  • 1
  • 7