12

Sometimes we need to drain nodes in Kubernetes. When I manually set up a k8s cluster, I can drain the specific node then terminate that machine. While in EKS, nodes are under auto scaling group, which means I can't terminate a specific instance(node). If I manually terminate a instance, another instance(node) will be automatically added into eks cluster.

So is there any suggested method to drain a node in EKS?

Shengxin Zhang
  • 131
  • 1
  • 1
  • 4
  • What are you asking here? `kubectl drain` will drain a Node, ASG or not. Terminating an instance will _effectively_ drain it, although not in the way you'd like. And yes, adding instances in place of other instances sounds like what an ASG was designed to do. If you don't want a replacement, perhaps scale the ASG down to prevent that behavior? – mdaniel Aug 14 '18 at 06:09
  • Currently if I want drain a node in EKS. I am going to: 1. run "kubectl drain xxx"; 2. terminate the ec2 instance drained in last step; 3. modify the desired instance number in ASG. While after these steps, ASG will terminate another instance(not the one i drained) and start a new instance. So after all, 2 instances are terminated and one new instance started, which is not what I want since I only want to drain & terminate one node(instance). – Shengxin Zhang Sep 10 '18 at 18:28
  • Well, if I'm hearing you correctly, you're conflating two separate things: you want `node-1` drained and terminated, and the steps I outlined will 100% do that. But the other half of your question is somehow related to the sizing of the ASG (which is not related to draining anything). Please do consider updating your question to be more specific. – mdaniel Sep 10 '18 at 20:55

2 Answers2

34

These steps should work:

  1. kubectl get nodes
  2. kubectl cordon <node name>
  3. kubectl drain <node name> --ignore-daemonsets or
    kubectl drain <node name> --ignore-daemonsets --delete-emptydir-data
  4. aws autoscaling terminate-instance-in-auto-scaling-group --instance-id <instance-id> --should-decrement-desired-capacity

For AWS autoscaling group, if you have nodes span out to multiple zones, consider delete nodes in each zones instead of all nodes from a single zone.

After the execution of the above commands, check the autoscaling group's desired number. It should decrease automatically. If you are using Terraform or another automation framework, don't forget to update your autoscaling group config in your infrastructure script.

Oleksandr Shmyrko
  • 1,720
  • 17
  • 22
Steve-Liang
  • 441
  • 4
  • 6
2
1.) kubectl get nodes

2.) kubectl cordon <node name>

3.) kubectl drain <node name> --ignore-daemonsets --delete-emptydir-data

Flag --delete-local-data has been deprecated, This option is deprecated and will be deleted. Use --delete-emptydir-data.

yasin lachini
  • 5,188
  • 6
  • 33
  • 56