StatefulSet deployment
Is it possible to force kubernetes deployment to complete and deploy 2
pods, even when the readiness probe is still not passing?
Assuming it's meant statefulSet instead of deployment as object, the answer is no, it's not possible by design, most important is second point:
- For a StatefulSet with N replicas, when Pods are being deployed, they are created sequentially, in order from {0..N-1}.
- Before a scaling operation is applied to a Pod, all of its predecessors must be Running and Ready.
- When Pods are being deleted, they are terminated in reverse order, from {N-1..0}.
When the nginx example above is created, three Pods will be deployed
in the order web-0, web-1, web-2. web-1 will not be deployed before
web-0 is Running and Ready, and web-2 will not be deployed until web-1
is Running and Ready
StatefulSets - Deployment and scaling guaranties
Readyness probe, endpoints and potential workaround
If the readiness probe failure, always prevent the deployment, Is
there other way to selectively expose only ready pods in the load
balancer, while not marking them as Unready during the deployment?
This is by design, pods are added to service endpoints once they are in ready state.
Some kind of potential workaround can be used, at least in simple example it does work, however you should try and evaluate if this approach will suit your case, this is fine to use as initial deployment.
statefulSet can be started without readyness probe included, this way statefulSet will start pods one by one when previous is run and ready, liveness may need to set up initialDelaySeconds so kubernetes won't restart the pod thinking it's unhealthy. Once statefulSet is fully run and ready, you can add readyness probe to the statefulSet.
When readyness probe is added, kubernetes will restart all pods again starting from the last one and your application will need to start again.
Idea is to start all pods and they will be able to serve requests +- at the same time, while with readyness probe applied, only one pod will start in 5 minutes for instance, next pod will take 5 minutes more and so on.
Example
Simple example to see what's going on based on nginx webserver and sleep 30 command which makes kubernetes think when readyness probe is setup that pod is not ready.
- Apply
headless service
- Comment
readyness probe in statefulSet and apply manifest
- Observe that all pods are created right after previous pod is
running and ready
- Uncomment
readyness probe and apply manifest
- Kubernetes will recreate all pods starting from the last one waiting this time
readyness probe to complete and flag a pod as running and ready.
Very convenient to use this command to watch for progress:
watch -n1 kubectl get pods -o wide
nginx-headless-svc.yaml:
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: nginx
spec:
ports:
- port: 80
name: web
clusterIP: None
selector:
app: nginx
nginx-statefulset.yaml:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: nginx
spec:
selector:
matchLabels:
app: nginx
serviceName: "nginx"
replicas: 3
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
name: web
command: ["/bin/bash", "-c"]
args: ["sleep 30 ; echo sleep completed ; nginx -g \"daemon off;\""]
readinessProbe:
tcpSocket:
port: 80
initialDelaySeconds: 1
periodSeconds: 5
Update
Thanks to @jesantana for this much easier solution.
If all pods have to be scheduled at once and it's not necessary to wait for pods readyness, .spec.podManagementPolicy can be set to Parallel. Pod Management Policies
Useful links:
