0

I've investigated a couple of helm charts for apache spark deployment and found that most of them use statefulset for deployment instead of normal k8s deployment resource.

E.g. the microsoft/spark uses normal deployment while bitnami/spark prefers statefulset.

I am just wondering is there any specific reason to do that?

Russell Bie
  • 341
  • 2
  • 11

1 Answers1

1

Apache Spark is a stateful service, those should be deployed as StatefulSet.

Only stateless services should be deployed as Deployment. Applications that are stateless follow the Twelve Factor App principles. Making an app stateless make it much easier to run as a distributed system e.g. multiple instances in Kubernetes. But not everything can be stateless, and StatefulSet is a an option for stateful services.

Jonas
  • 121,568
  • 97
  • 310
  • 388
  • But there is still some spark helm chart such as microsoft/spark which uses normal deployment. https://hub.helm.sh/charts/microsoft/spark – Russell Bie May 16 '20 at 10:24