0

I was trying to resize persistent volumes associated with a stateful set today. I am using Azure Kubernetes service v1.26.6. The persistent voluem is created from a storage class of type "default".

As per the official Kubernetes documentation at:https://kubernetes.io/blog/2022/05/05/volume-expansion-ga/, it is now possible to expand the size of a persistent volume without any downtime, just by updating the spec field of the PVC of the stateful set and re-deploying the stateful set. However, i attempted the same today(i.e tried to redeploy an already deployed stateful set but with increased size for the PVC) and i ran into the usual error:

Error: UPGRADE FAILED: cannot patch "grafana-loki-querier" with kind StatefulSet: StatefulSet.apps "grafana-loki-querier" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

To overcome this error, i then followed the instructions at: https://www.giffgaff.io/tech/resizing-statefulset-persistent-volumes-with-zero-downtime and was then finally able to successfully deploy the helm chart containing updated sizes for the resized PVC.

Questions -Am i misunderstanding the official Kubernetes documentation that i have linked in the question?

-I am curious. Technically, if the size of the PVC can be increased by editing the spec field in the PVC object directly(kubectl edit pvc), why is Kubernetes imposing a restriction to do the same by directly updating the size in the PVC manifest files ie. why cannot i increase the size of the PVC using a helm deployment?

Why is the workflow not an integral part of Kubernetes deployments using Helm?

Why does it involve deleting the stateful set followed by recreation of the stateful set? What is the reason for the same?

enter image description here

enter image description here

Kiran Hegde
  • 680
  • 4
  • 14

1 Answers1

1

Questions -Am i misunderstanding the official Kubernetes documentation that i have linked in the question?

Not really, but you are only partially true.

Even if StorageClass supports resizing, Kubernetes doesn't support (yet) volume expansion through StatefulSets. Please refer to the official enhancement issue at Support Volume Expansion Through StatefulSets and Kubernetes StatefulSet: support resize pvc storage.

More Insights

Why is the workflow not an integral part of Kubernetes deployments using Helm?

In this case, this has nothing to do with helm charts but actually with the Kubernetes controller involved in the application stack, as the controller is continuously monitoring the app and making the required changes as per its defined manifest and hence as mentioned in your workaround document you had to scale it to 0

Second, we had to delete the stateful state with the orphan strategy so that the actual pod ( working application ) does not get deleted and only the stateful set controller(kind) is deleted. Because of that, you are allowed to Update the PVC and then it gets resized only if resizing is supported and enabled on your storageClass->Storage Driver which seems the case.

Normally helm has quite a lot of flaws but in this case, it's not the culprit.

ishuar
  • 1,193
  • 1
  • 3
  • 6
  • Hello @ishuar Thanks for your response. I guess i should have been more clear. I was curious to know as to why the stateful set controller itself doesn't codify the workflow that is currently recommended as a workaround. The moment there is a change in the size of the stateful set in the specification for a stateful set, the workflow can be triggered. – Kiran Hegde Aug 21 '23 at 05:01
  • @KiranHegde, unfortunately, that's the way how it is with stateful sets at the moment, `volumeClaimTemplates` is an immutable field. Every Kubernetes user is expecting/wishing for this feature for a long time. You can refer to the issues attached in my answer. – ishuar Aug 21 '23 at 21:29