Kubernetes Statefulsets: Restart all pods concurrently (instead of in sequence)

Question

I have a use-case for concurrent restart of all pods in a statefulset.

Does kubernetes statefulset support concurrent restart of all pods?

According to the statefulset documentation, this can be accomplished by setting the pod update policy to parallel as in this example:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql-db
spec:
  podManagementPolicy: Parallel
  replicas: 3

However this does not seem to work in practice, as demonstrated on this statefulset running on EKS:

Apply this:

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: producer
  namespace: ragnarok
spec:
  selector:
    matchLabels:
      app: producer
  replicas: 10
  podManagementPolicy: "Parallel"
  serviceName: producer-service
  template:
    metadata:
      labels:
        app: producer
    spec:
      containers:
      - name: producer
        image: archbungle/load-tester:pulsar-0.0.49
        imagePullPolicy: IfNotPresent

Rollout restart happens in sequence as if disregarding the rollout policy setting:


(base) welcome@Traianos-MacBook-Pro eks-deploy % kubectl get pods -n ragnarok | egrep producer
producer-0                               1/1     Running             0          3m58s
producer-1                               1/1     Running             0          3m56s
producer-2                               1/1     Running             0          3m53s
producer-3                               1/1     Running             0          3m47s
producer-4                               1/1     Running             0          3m45s
producer-5                               1/1     Running             0          3m43s
producer-6                               1/1     Running             1          3m34s
producer-7                               0/1     ContainerCreating   0          1s
producer-8                               1/1     Running             0          16s
producer-9                               1/1     Running             0          19s

(base) welcome@Traianos-MacBook-Pro eks-deploy % kubectl get pods -n ragnarok | egrep producer
producer-0                               1/1     Running       0          4m2s
producer-1                               1/1     Running       0          4m
producer-2                               1/1     Running       0          3m57s
producer-3                               1/1     Running       0          3m51s
producer-4                               1/1     Running       0          3m49s
producer-5                               1/1     Running       0          3m47s
producer-6                               0/1     Terminating   1          3m38s
producer-7                               1/1     Running       0          5s
producer-8                               1/1     Running       0          20s
producer-9                               1/1     Running       0          23s

(base) welcome@Traianos-MacBook-Pro eks-deploy % kubectl get pods -n ragnarok | egrep producer
producer-0                               1/1     Running       0          4m8s
producer-1                               1/1     Running       0          4m6s
producer-2                               1/1     Running       0          4m3s
producer-3                               1/1     Running       0          3m57s
producer-4                               1/1     Running       0          3m55s
producer-5                               0/1     Terminating   0          3m53s
producer-6                               1/1     Running       0          4s
producer-7                               1/1     Running       0          11s
producer-8                               1/1     Running       0          26s
producer-9                               1/1     Running       0          29s

`kubectl -n ragnarok delete po -l app=producer` will do that, if I understand your question correctly — mdaniel, Feb 20 '22 at 17:51

Oguzhan Aygun · Accepted Answer · 2022-02-21T00:34:51.907

As the document pointed, Parallel pod management will effective only in the scaling operations. This option only affects the behavior for scaling operations. Updates are not affected.

Maybe you can try something like kubectl scale statefulset producer --replicas=0 -n ragnarok and kubectl scale statefulset producer --replicas=10 -n ragnarok

According to documentation, all pods should be deleted and created together by scaling them with the Parallel policy.

Reference : https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#parallel-pod-management

Kubernetes Statefulsets: Restart all pods concurrently (instead of in sequence)

1 Answers1