Node had taints that the pod didn't tolerate error when deploying to Kubernetes cluster

Question

I am trying to deploy my microservices into Kubernetes cluster. My cluster having one master and one worker node. I created this cluster for my R&D of Kubernetes deployment. When I am trying to deploy I am getting the even error message like the following,

Events:
 Type     Reason            Age        From               Message
  ----     ------            ----       ----               -------
 Warning  FailedScheduling  <unknown>  default-scheduler  0/2 nodes are available: 2 node(s) had taints that the pod didn't tolerate

My attempt

When I am exploring about the error, I found some comments in forums for restarting the docker in the node etc. So after that I restarted Docker. But still the error is the same.

When I tried the command kubectl get nodes it showing like that both nodes are master and both are ready state.

NAME           STATUS   ROLES    AGE     VERSION
 mildevkub020   Ready    master   6d19h   v1.17.0
 mildevkub040   Ready    master   6d19h   v1.17.0

I did not found worker node here. I created one master (mildevkub020) and one worker node (mildev040) with one load balancer. And I followed the official documentation of Kubernetes from the following link,

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/

My question

Is this error is because of the cluster problem? Because I am not finding the cluster worker node. Only master node.

The answer has been given to you. https://stackoverflow.com/a/59491824/6064025 — Aamir M Meman, Dec 27 '19 at 15:25

score 52 · Accepted Answer · answered Dec 26 '19 at 09:24

52

You can run below command to remove the taint from master node and then you should be able to deploy your pod on that node

kubectl taint nodes  mildevkub020 node-role.kubernetes.io/master-
kubectl taint nodes  mildevkub040 node-role.kubernetes.io/master-

Now regarding why its showing as master node check the command you ran to join the node with kubeadm. There are separate commands for master and worker node joining.

answered Dec 26 '19 at 09:24

Arghya Sadhu

41,002
9
78
107

5

Thanks! To remove taint from **all** master nodes: `for node in $(kubectl get nodes --selector='node-role.kubernetes.io/master' | awk 'NR>1 {print $1}' ) ; do kubectl taint node $node node-role.kubernetes.io/master- ; done` – Noam Manos Sep 01 '21 at 15:00
5

when i dod that i get `error: taint "node-role.kubernetes.io/master" not found` – scavenger Feb 17 '22 at 01:02

Brad Parks · Answer 2 · 2020-03-23T23:28:07.693

11

You can also get this "taint" type of message when your docker environment doesn't have enough resources allocated.

For example, in Docker Desktop for Mac, allocate more memory/cpu/swap in preferences, and it may solve your problem.

This can also happen if kubernetes auto scaling doesn't have enough nodes to launch a new pod, which you will may see as "Insufficient CPU" on describe.

edited Mar 23 '20 at 23:28

answered Mar 11 '20 at 12:59

Brad Parks

66,836
64
257
336

2

Had this same "taint" events when configuring ingress-nginx for mac for the first time. The ingress controller got stuck in Pending state. Doubling up memory and swap allocation solved it. – Alexander Fradiani Jun 29 '20 at 09:04

score 6 · Answer 3 · answered Jun 08 '22 at 09:44

6

Remove the taints on the master so that you can schedule pods on it.

kubectl taint nodes --all node-role.kubernetes.io/master-

It should return the following.

node/<your-hostname> untainted

answered Jun 08 '22 at 09:44

elmerzouki

199
2
7

score 4 · Answer 4 · edited Jul 10 '22 at 05:10

4

In k8s version 1.24.2, you have to run below command:

kubectl taint nodes <node_name> dedicated=special-user:NoSchedule

edited Jul 10 '22 at 05:10

Adrian Mole

49,934
160
51
83

answered Jul 05 '22 at 14:03

wei_elsa

41
2

4

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Jul 06 '22 at 04:32

score 2 · Answer 5 · answered Feb 14 '21 at 11:46

The same issue I had faced because my Kubernetes worker node was down(turned off)

Manjunath-MacBook-Air:manjunath$ kubectl get nodes
NAME                       STATUS     ROLES   AGE   VERSION
aks-agentpool-****-*   NotReady   agent   31d   v1.18.10

After starting the VM(Kubernetes worker instance ), the issue got resolved

Manjunath-MacBook-Air: manjunath$ kubectl get nodes
NAME                       STATUS   ROLES   AGE   VERSION
aks-agentpool-****-*   Ready    agent   31d   v1.18.1

score 2 · Answer 6 · answered Jul 27 '21 at 14:39

I get this with microk8s when I reboot the machine that my "cluster" runs on. Enough of microk8s comes back online to convince me that it's "up", but pods get stuck pending with this error.

I just have to run microk8s start and whatever is stuck gets unstuck (until the next reboot).

score 2 · Answer 7 · answered Jul 25 '22 at 08:28

Put the name of the node that gives us this error : FailedScheduling

kubectl taint nodes <name-node-master> node-role.kubernetes.io/control-plane:NoSchedule-

example :

kubectl taint nodes prod-k8s-master.octopeek-dns.com node-role.kubernetes.io/control-plane:NoSchedule-

score 2 · Answer 8 · answered Aug 14 '22 at 16:16

In Kubernetes 1.24+ you need to run this command to allow scheduling apps on the master/control-plane nodes:

kubectl taint nodes --all node-role.kubernetes.io/control-plane- node-role.kubernetes.io/master-

Ref: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#control-plane-node-isolation

score 1 · Answer 9 · answered Apr 10 '23 at 18:08

1

This is what worked for me on Kubernetes 1.26.x :

kubectl taint nodes mildevkub020 node-role.kubernetes.io/control-plane:NoSchedule-

answered Apr 10 '23 at 18:08

secavfr

628
1
9
24

score 0 · Answer 10 · answered Apr 26 '21 at 15:23

I found this in docs.

https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/

Keep in mind that setting up your cluster to use cloud controller manager will change your cluster behaviour in a few ways:

kubelets specifying --cloud-provider=external will add a taint node.cloudprovider.kubernetes.io/uninitialized with an effect NoSchedule during initialization. This marks the node as needing a second initialization from an external controller before it can be scheduled work. Note that in the event that cloud controller manager is not available, new nodes in the cluster will be left unschedulable. The taint is important since the scheduler may require cloud specific information about nodes such as their region or type (high cpu, gpu, high memory, spot instance, etc).

score 0 · Answer 11 · answered May 15 '23 at 06:02

For Kubernetes version v1.25.5, in my case I've added the following tolerations to my deployments

apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  namespace: <namespace>
spec:
  replicas: 2
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
        - name: backend
          image: <image_name>
          ports:
            - containerPort: 8081
      tolerations:
        - key: <node_key> # IMPORTANT here
          operator: Equal
          value: "true"
          effect: NoSchedule

To get nodes taint keys, use the following command:

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{.spec.taints}{"\n\n"}{end}'

Update key, operator, value, effect to meet with node taints.

score 0 · Answer 12 · answered May 18 '23 at 15:33

0

these commands work with all nodes for control-plane

kubectl taint nodes --all node-role.kubernetes.io/control-plane-

for master

kubectl taint nodes --all node-role.kubernetes.io/master-

answered May 18 '23 at 15:33

elmerzouki

199
2
7

score -1 · Answer 13 · answered Mar 23 '22 at 03:56

-1

I faced the same issue a while ago and had to add tolerations to overcome this.

tolerations:
  - key: "os"
    operator: "Equal"
    value: "windows"
    effect: "NoSchedule"

answered Mar 23 '22 at 03:56

Gnani rahul Nutakki

1

Node had taints that the pod didn't tolerate error when deploying to Kubernetes cluster

13 Answers13

Linked