Kubernetes Mongo with persistent volume FailedScheduling 0/1 nodes are available: 1 node(s) didn't match node selector

Question

I tried to follow this link about Kubernetes MongoDB with persistent volume (exactly same).

but aftrer I deploy everything and take a look on the pod, I got these errors:

Name:           mongodb-standalone-0
Namespace:      default
Priority:       0
Node:           <none>
Labels:         app=database
                controller-revision-hash=mongodb-standalone-7688499856
                selector=mongodb-standalone
                statefulset.kubernetes.io/pod-name=mongodb-standalone-0
Annotations:    <none>
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  StatefulSet/mongodb-standalone
Containers:
  mongodb-standalone:
    Image:      mongo:4.0.8
    Port:       <none>
    Host Port:  <none>
    Environment:
      MONGO_INITDB_ROOT_USERNAME_FILE:  /etc/k8-training/admin/MONGO_ROOT_USERNAME
      MONGO_INITDB_ROOT_PASSWORD_FILE:  /etc/k8-training/admin/MONGO_ROOT_PASSWORD
    Mounts:
      /config from mongodb-conf (ro)
      /data/db from mongodb-data (rw)
      /docker-entrypoint-initdb.d from mongodb-scripts (ro)
      /etc/k8-training from k8-training (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-xbl5z (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  k8-training:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  k8-training
    Optional:    false
  mongodb-scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mongodb-standalone
    Optional:  false
  mongodb-conf:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mongodb-standalone
    Optional:  false
  mongodb-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  mongodb-standalone
    ReadOnly:   false
  default-token-xbl5z:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-xbl5z
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  kubernetes.io/hostname=mongodb-node
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age        From               Message
  ----     ------            ----       ----               -------
  Warning  FailedScheduling  <unknown>  default-scheduler  0/1 nodes are available: 1 node(s) didn't match node selector.
  Warning  FailedScheduling  <unknown>  default-scheduler  0/1 nodes are available: 1 node(s) didn't match node selector.

Result of kubectl get nodes --show-labels:

minikube   Ready    master   43h   v1.17.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=minikube,kubernetes.io/os=linux,node-role.kubernetes.io/master=

I've tried to find a way to debug this issue, got nothing..

I successfully run my MongoDB with my apps in Kubernetes, but the problem is I want to have persistent volume for my data and so far I couldn't find the right approach to make it works. I appreciate any help, thank you.

UPDATE

I have followed to change the nodeSelector, but still got same error:

Name:           mongodb-standalone-0
Namespace:      default
Priority:       0
Node:           <none>
Labels:         app=database
                controller-revision-hash=mongodb-standalone-74895d955f
                selector=mongodb-standalone
                statefulset.kubernetes.io/pod-name=mongodb-standalone-0
Annotations:    <none>
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  StatefulSet/mongodb-standalone
Containers:
  mongodb-standalone:
    Image:      mongo:4.0.8
    Port:       <none>
    Host Port:  <none>
    Environment:
      MONGO_INITDB_ROOT_USERNAME_FILE:  /etc/k8-training/admin/MONGO_ROOT_USERNAME
      MONGO_INITDB_ROOT_PASSWORD_FILE:  /etc/k8-training/admin/MONGO_ROOT_PASSWORD
    Mounts:
      /config from mongodb-conf (ro)
      /data/db from mongodb-data (rw)
      /docker-entrypoint-initdb.d from mongodb-scripts (ro)
      /etc/k8-training from k8-training (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-xbl5z (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  k8-training:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  k8-training
    Optional:    false
  mongodb-scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mongodb-standalone
    Optional:  false
  mongodb-conf:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mongodb-standalone
    Optional:  false
  mongodb-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  mongodb-standalone
    ReadOnly:   false
  default-token-xbl5z:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-xbl5z
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  kubernetes.io/hostname=minikube
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age        From               Message
  ----     ------            ----       ----               -------
  Warning  FailedScheduling  <unknown>  default-scheduler  0/1 nodes are available: 1 node(s) didn't find available persistent volumes to bind.
  Warning  FailedScheduling  <unknown>  default-scheduler  0/1 nodes are available: 1 node(s) didn't find available persistent volumes to bind.

there, have added more explanation about the ouput of kubectl describe pod — Darryl RN, Feb 09 '20 at 16:22
now I got this error `0/1 nodes are available: 1 node(s) didn't find available persistent volumes to bind.`.. but I couldn't find in which object of sts, the persistent volume was defined.. — Darryl RN, Feb 09 '20 at 17:04
I have edited my answer..try that...you have a persistent volume which needs modification too — Arghya Sadhu, Feb 09 '20 at 17:11
still not working.. hv changed pv to `- key: kubernetes.io/hostname=minikube`, but it doesn't work, couldn't add `=` in that string. — Darryl RN, Feb 09 '20 at 17:20
you should not use = in pv..you need to configure it using operator...check my answer — Arghya Sadhu, Feb 09 '20 at 17:24
oohhh.. right, its the `values` that is different.. my bad.. let me try, will let u know asap — Darryl RN, Feb 09 '20 at 17:27
I've successfully run the sts, but when I test the persistent volume, it shows warning: `Warning: unable to run listCollections, attempting to approximate collection names by parsing connectionStatus`.. what is wrong this time? — Darryl RN, Feb 09 '20 at 17:46
after I delete sts and re-apply it, collections not appear as expected.. how to debug this? — Darryl RN, Feb 09 '20 at 17:49
I guess that's a Mongodb error...I suggest asking a different question with all details.As far as this question is concerned its answered. — Arghya Sadhu, Feb 09 '20 at 17:50
I'm sorry, currently I handle another task, will post my yaml files once I get back to Kubernetes task again. thanks for your help — Darryl RN, Feb 11 '20 at 20:36

score 2 · Answer 1 · answered Feb 09 '20 at 16:34

I guess you are missing label kubernetes.io/hostname: mongodb-node on your node.

you can remove nodeSelector from your yaml:

      nodeSelector:
        kubernetes.io/hostname: mongodb-node

you can label your node with kubectl label node <your_node_name> kubernetes.io/hostname=mongodb-node --overwrite but I do not recommend the this approach.
you can change nodeSelector to a proper one, juest check your kubernetes.io/hostname with kubectl get no --show-labels

This is one of default Kubernetes label, I don't know what consequences it might have - it may break your cluster. — FL3SH, Feb 09 '20 at 19:00

Arghya Sadhu · Answer 2 · 2020-02-09T17:10:51.230

1

Change the nodeselector in your deployment to kubernetes.io/hostname=minikube

Edit:

In your persistent volume you have a nodeAffinity which you need to modify and provide it correct value

nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
            - minikube

edited Feb 09 '20 at 17:10

answered Feb 09 '20 at 16:35

Arghya Sadhu

41,002
9
78
107

Kubernetes Mongo with persistent volume FailedScheduling 0/1 nodes are available: 1 node(s) didn't match node selector

UPDATE

2 Answers2