Kubernetes Startup probe when endpoint is not reachable

Question

I have the following deployment config. The test-worker-health and health endpoints are both unreachable as the application is failing due to an error. The startup probe keeps restarting the container after failing as restartPolicy: Always. The pods enter CrashLoopBackoff state. Is there a way to fail such startup probe?

livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8080
          periodSeconds: 20
          successThreshold: 1
          timeoutSeconds: 30
startupProbe:
          httpGet:
            path: /test-worker-health
            port: 8080
          failureThreshold: 12
          periodSeconds: 10

Jonas · Accepted Answer · 2020-09-01T17:33:54.317

2

The startup probe keeps restarting the container after failing

The startupProbe does not restart your container, but the livenessProbe does.

The pods enter CrashLoopBackoff state. Is there a way to fail such startup probe?

If you remove the livenessProbe, you will not get this restart-behavior. You may want to use a readinessProbe instead?

Is there a way to fail such startup probe?

What do you mean? It is already "failing" as you say. You want automatic rollback? That is provided by e.g. Canary Deployment, but is a more advanced topic.

edited Sep 01 '20 at 17:33

answered Sep 01 '20 at 17:25

Jonas

121,568
97
310
388

thanks jonas! I will try it out. But I have a doubt. As per https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/, startup probe disables liveness and readiness checks until it succeeds, making sure those probes don't interfere with the application startup. So isn't the liveness probe deferred to go after startup? – Ruchika Salwan Sep 01 '20 at 17:35
yes I wanted a rollback to a previous deployment. If startup probe is not restarting the container, it should fail the pods too and the previous deployment should work – Ruchika Salwan Sep 01 '20 at 17:39
Yes, livenessProbe is deferred to after startupProbe, but the container may crash before. – Jonas Sep 01 '20 at 18:02
"If startup probe is not restarting the container, it should fail the pods too" - what do you mean with "fail the pods too", I think there is no such thing? To do rollback of the `Deployment` you have to add or implement features for that - it does not come out of the box. You may use `ReadinessProbe` to only allow traffic when the pod is healthy - but other than that you need to add with other stuff. – Jonas Sep 01 '20 at 18:05
If you have multiple replicas and performing a Rolling Deployment, it will not continue with the next replica if the first fails. https://www.bluematador.com/blog/kubernetes-deployments-rolling-update-configuration – Jonas Sep 01 '20 at 18:08
1

thanks Jonas! I would be accompanying the startup probe with a readiness probe like you said. Also recently this was logged as a bug by another user and it was converted into a new feature - https://github.com/kubernetes/kubernetes/pull/92196 – Ruchika Salwan Sep 02 '20 at 19:33

score 0 · Answer 2 · answered Mar 04 '22 at 11:03

According to your configuration, the startupProbe is tried within 120seconds after which it fails if it doesn't succeed atleast once during that period.

If your application requires more time to start up .i.e > 120seconds, then the startupProbe would keep restarting your application before it's booted up completely.

I'd suggest increasing the failureThreshold to afford your application sufficient time to boot-up.

Kubernetes Startup probe when endpoint is not reachable

2 Answers2