3

We are deploying multiple Bitnami/MongoDB via helm on an AKS Cluster running on Kubernetes 1.19.7 on 4 Nodes (DS2_V2).

Each MongoDb is installed separately via helm install, however, completely randomically one of them fails and go in timedout condition reporting:

Unable to attach or mount volumes: unmounted volumes=[datadir], unattached volumes=[datadir-db-token-p55m7]: timed out waiting for the condition

This is totally random and occuring since last week. As Storage Class we use the default from aks, based on azure-storage-disk. We have tried to change the volume binding move from WaitingForFirstConsumer to Immediate without success.

The full list of event is:

  1. Successfully assigned tp-xxxxxx/xxxxxx-api-db-0 to aks-default-26759343-vmss000003
  2. AttachVolume.Attach succeeded for volume "pvc-adf7e23f-2264-4eb5-bc29-9e1707a3f5a9"
  3. Unable to attach or mount volumes: unmounted volumes=[datadir], unattached volumes=[datadir xxxxx-api-db-token-p55m7]: timed out waiting for the condition
  4. MountVolume.WaitForAttach failed for volume "pvc-adf7e23f-2264-4eb5-bc29-9e1707a3f5a9" : timed out waiting for the condition 5.Unable to attach or mount volumes: unmounted volumes=[datadir], unattached volumes=[xxxx-api-db-token-p55m7 datadir]: timed out waiting for the condition

This lead to helm install to reach the timeout and then installation fails.

Slevin
  • 141
  • 1
  • 8
  • Is your pod in the same region as the PV? You may need to add affinity to fix it or use an external storage. – rohatgisanat Mar 20 '21 at 16:39
  • The cluster is located in West-Europe as the agent pool – Slevin Mar 20 '21 at 16:51
  • Your Issue seems to be same as https://github.com/kubernetes/kubernetes/issues/94764. Can you please check in kubelet logs if the Persistent Volume was mounted after the timeout ? Please keep the volumeBindingMode = WaitingForFirstConsumer for the storage class otherwise your pv may get created in a different zone than the pod and cannot be mounted into the pod leading to unschedulable Pods. – Sagar Velankar Mar 21 '21 at 14:34
  • Can it be you have too many volumes attached to the same instance? Azure virtual machines have a limit on how many disks can be attached at the same time. In case of `DS2_V2` this limit is 8 – whites11 Mar 22 '21 at 19:35
  • Can you share the chart and the YAML file you use to create the PV? – Charles Xu Mar 23 '21 at 08:02
  • @Charles Xu We are installing multiple instances of Bitnami/MongoDB https://github.com/bitnami/charts/tree/master/bitnami/mongodb where we set architecture=replicaSet . – Slevin Mar 23 '21 at 09:36
  • You need to share more details about the PV you use to mount. – Charles Xu Mar 24 '21 at 06:36
  • Got the same issue, have you solved it @Slevin? – Goon Nguyen Aug 03 '21 at 02:23
  • 1
    @GoonNguyen we started creating AKS cluster with 3 zones (we were just using one) and while in contact with Microsoft seems that when this post was made there were some bug/issue broadly on AKS. Since then we never experience the issue again. – Slevin Aug 19 '21 at 15:22

0 Answers0