2

In a Azure AKS kubernetes cluster, after a cluster version upgrade the nodepool nodes, I have a PV that has this node affinity:

Node Affinity:
  Required Terms:
    Term 0:        failure-domain.beta.kubernetes.io/region in [westeurope]

The nodes don't have the label so the Deployment creates a Pod that cannot be scheduled for the corresponding PVC for this PV. The Pod is never started:

  Warning  FailedScheduling  15m    default-scheduler  0/3 nodes are available: 3 node(s) had volume node affinity conflict. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling..

How can I add the label to the node or remove the label from the PV? I have tried to add the label to the node but I get:

Error from server: admission webhook "aks-node-validating-webhook.azmk8s.io" denied the request: (UID: 931bf139-1579-4e96-b164-e4e6e2fdae65) User is trying to add or update a non-changeable system label (failure-domain.beta.kubernetes.io/region:westeurope). This action is denied..

Is the only solution to backup and restore the PV into a new one that does not have that deprecated label? What would the best process to do it (or any alternative solution)

icordoba
  • 1,834
  • 2
  • 33
  • 60

1 Answers1

1

We had the same problem. How we resolved it:

  1. Copy output from "kubectl get pvc" to get the link between the pvc and the pv.
  2. Locate the disk in azure portal and create a snapshot of the disk. (In the MC_ resource group to the aks
  3. Edit deployment in kubernetes and set replicacout to 0. Save and see that the pod are stopping ang removed.
  4. Delete the PVC for this pod.
  5. Edit deployment in kubernetes and set replicacout to 1. Save and see that there is a new PVC and a new PV created.
  6. Edit deployment again and set replicacount to 0.
  7. Locate the new disk in azure portal. Use "kubectl get pvc" to locate.
  8. Delete new disk in azure portal
  9. Locate snapshot created in pt 2.
  10. Create a new disk based on the snapshot. New disk should have the same name as the disk deleted in pt 6.
  11. Edit deployment in kubernetes and set replicacount to 1. It should now start using the old disk with the new pvc and pv.

Take backup of what ever referances and disk you can before starting

user21907706
  • 11
  • 1