Safely unbind statefulset/my-pod-0 pvc without affecting the other replicas

Question

I have a statefulset with 2 pods, running on separate dedicated nodes, each with its own pvc.

I need to perform some maintenance to the PVs assigned to each of the PVCs assigned to the statefulset's pods.

So far I've been able to scale down the statefulset to 1 replica, which brought offline statefulset/pod-1, this way I've been able to perform the maintenance task.

Now I need to perform the same task to statefulset/pod-0, without taking offline statefulset/pod-1.

What are my options? Remember, statefulset/pod-0's pv must be unmounted in order to start the maintenance task.

what task do you want to perform on your PVC that requires it to be unbound? — OlGe, Jun 25 '22 at 07:44
@OlGe i need to move the cinder pv from one backend to another, I corrected the question, I need it to be unmounted not unbound — J11, Jun 25 '22 at 08:06

score 3 · Accepted Answer · answered Jul 06 '22 at 16:41

3

I was able to perform the task on statefulset/pod-0 by:

Cordon the node
Delete statefulset/pod-0

Once the task was complete uncordon the node and the pod started automatically without any issues.

This will only work if the pod contains a specific nodeAffinity

answered Jul 06 '22 at 16:41

J11

426
1
6
17

creative solution! nice! – OlGe Aug 30 '22 at 12:21
It didn't work for my situation but thats really a good idea – user1279741 Jan 25 '23 at 19:49

score 2 · Answer 2 · answered Jun 25 '22 at 08:35

I don't think that this is possible to achieve with one statefulset because of the Deployment and Scaling Guarantees a statefulset provides. To unmount the volume of a pod, you must shutdown/delete the pod first and before this can be done to a pod in a statefulset, "all of its successors must be completely shutdown."

My advice is to plan and communicate a maintenance window with the stakeholders of the application and scale down the statefulset entirely to apply the changes to the volume in your storage backends. Moving volumes across storage backends is not a task to perform on a regular basis, so I think it is reasonable to ask for a one-time maintenance to do so.

It would be nice if Kubernetes offered a way to down scale the replicas starting from pod-0 up to pod-N — J11, Jun 25 '22 at 09:40

score 1 · Answer 3 · answered Jun 25 '22 at 09:06

1

...to move the cinder pv from one backend to another

With Cinder CSI you can perform a snapshot or clone the volume into a new PV and move to another.

answered Jun 25 '22 at 09:06

gohm'c

13,492
1
9
16

Would I need to shut down statefulset/pod-0 prior to the clone operation? – J11 Dec 04 '22 at 18:24

Safely unbind statefulset/my-pod-0 pvc without affecting the other replicas

3 Answers3