2

I have the following scenario:

  1. A StatefulSet with 1 replica
  2. Update the template section and scale it in the same operation using helm as application manager
  3. The order of operation is the following:
    1. Scaling to 3
    2. Update the replica with name 0

Because I cannot control to first update and after that scale, I am losing data because there is a specific logic in the new statefulset template.

Is there a way to control the ordering of those operations?

The service in question is Redis, we are trying to get from standalone mode (1 replica) to replication(HA) without losing data.

Alin Alexandru Baicu
  • 21
  • 1
  • 2
    The recommended way is to use a k8s-operator for Redis which supports horizontal scaling. – Kamol Hasan Mar 09 '22 at 12:02
  • Welcome to the community. Any reasons why you can't switch logic to updating first replica and then scaling out? – moonkotte Mar 10 '22 at 08:37
  • Because helm is controlling that, the operation is like an atomic one and I do not want to first update and after that scale that means two operations to do. If I would have a param on the STS to control that I would be unblocked. – Alin Alexandru Baicu Mar 10 '22 at 11:32
  • @AlinAlexandruBaicu I'm not sure I understand you. Which param would you like to have? What exactly should it do? There's a flag [`.spec.updateStrategy`](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#update-strategies) and when it's set to `OnDelete` you will manually delete pods which you need to be recreated. But I'm not sure this is what you need. Could you please elaborate in more details exactly what you need? Otherwise it's done via helm and then you need to edit helm chart's logic. Or probably use operator as it was suggested in the first comment. – moonkotte Mar 11 '22 at 08:41
  • Let's go with an in-depth explanation, we have the following scenario: - Install helm chart Redis standalone basically single node – Alin Alexandru Baicu Mar 16 '22 at 08:47
  • - Redis has the following logic inside, the pod that is up is looking for a master, if we do not have a master then he is a master so, because the scale operation is executed first the replica-0 that hold the data is in place and replica-1 is going up it sees no master then he is the master and basically ignore all the data from replica-0. - After that sts continue with replica-2 as a slave and upgrade replica-0 as another slave. If the operation will be first update the replica-0 , elect it as master and after that scale to 3 with replica-1 , and replica-2 I would not loose any data. – Alin Alexandru Baicu Mar 16 '22 at 08:52
  • That's why I wanna control the order between scale and update on an STS. – Alin Alexandru Baicu Mar 16 '22 at 10:53

2 Answers2

0

For the moment I resolved the problem using a helm pre-install job that is basically scaling the sts to zero, after that helm is coming with the update.

Alin Alexandru Baicu
  • 21
  • 1
  • As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Mar 18 '22 at 23:34
0

I am not Redis expert, but I think that solution below should help you.

I would try to install another Redis HA instance (B) next to the existing one (A), taking as a data source for B a snapshot of A's PV. This could to avoid losing your data. For more information you can read more about volume snapshots.

See also this related problem.

Mikołaj Głodziak
  • 4,775
  • 7
  • 28