Why kubernetes reports "readiness probe failed" along with "liveness probe failed"

Question

I have a working Kubernetes deployment of my application.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  ...
  template:
    ...
    spec:
      containers:
      - name: my-app
        image: my-image
        ...
        readinessProbe:
          httpGet:
            port: 3000
            path: /
        livenessProbe:
          httpGet:
            port: 3000
            path: /

When I apply my deployment I can see it runs correctly and the application responds to my requests.

$ kubectl describe pod -l app=my-app

...
Events:
  Type    Reason     Age   From                                  Message
  ----    ------     ----  ----                                  -------
  Normal  Scheduled  4m7s  default-scheduler                     Successfully assigned XXX
  Normal  Pulled     4m5s  kubelet, pool-standard-4gb-2cpu-b9vc  Container image "my-app" already present on machine
  Normal  Created    4m5s  kubelet, pool-standard-4gb-2cpu-b9vc  Created container my-app
  Normal  Started    4m5s  kubelet, pool-standard-4gb-2cpu-b9vc  Started container my-app

The application has a defect and crashes under certain circumstances. I "invoke" such a condition and then I see the following in pod events:

$ kubectl describe pod -l app=my-app

...
Events:
  Type     Reason     Age               From                                  Message
  ----     ------     ----              ----                                  -------
  Normal   Scheduled  6m45s             default-scheduler                     Successfully assigned XXX
  Normal   Pulled     6m43s             kubelet, pool-standard-4gb-2cpu-b9vc  Container image "my-app" already present on machine
  Normal   Created    6m43s             kubelet, pool-standard-4gb-2cpu-b9vc  Created container my-app
  Normal   Started    6m43s             kubelet, pool-standard-4gb-2cpu-b9vc  Started container my-app
  Warning  Unhealthy  9s                kubelet, pool-standard-4gb-2cpu-b9vc  Readiness probe failed: Get http://10.244.2.14:3000/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy  4s (x3 over 14s)  kubelet, pool-standard-4gb-2cpu-b9vc  Liveness probe failed: Get http://10.244.2.14:3000/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
  Normal   Killing    4s                kubelet, pool-standard-4gb-2cpu-b9vc  Container crawler failed liveness probe, will be restarted

It is expected the liveness probe fails and the container is restarted. But why do I see Readiness probe failed event?

You have a simple problem. Either there is nothing serving on port 3000 on path / of your app or kubelet can't reach the port, so it is timing out. — suren, Oct 07 '19 at 13:47

Maksim Sorokin · Accepted Answer · 2019-10-08T12:43:00.157

4

As @suren wrote in the comment, readiness probe is still executed after container is started. Thus if both liveness and readiness probes are defined (and also fx they are the same), both readiness and liveness probe can fail.

Here is a similar question with a clear in-depth answer.

edited Oct 08 '19 at 12:43

answered Oct 08 '19 at 07:25

Maksim Sorokin

2,334
3
34
61

score 3 · Answer 2 · answered Oct 07 '19 at 12:14

3

The readiness probe is used to determine if the container is ready to serve requests. Your container can be running but not passing the probe. If it doesn't pass the check no service will redirect to this container.

By default the period of the readiness probe is 10 seconds.

You can read more here : https://docs.openshift.com/container-platform/3.9/dev_guide/application_health.html

answered Oct 07 '19 at 12:14

Paul

305
1
7

Thanks for the comment. But I have explicitly illustrated that the pod was running for more than 5 minutes without any readiness probe errors. Only after invoking application crash the readiness probe failure appears in the events. – Maksim Sorokin Oct 07 '19 at 12:33
1

Yes, at the beginning the pod is working correctly so both checks are OK, but when you crash the application the port 3000 is not available anymore (I guess), and since both checks are configured to check that port you see both errors in the events. The readiness probe is called every 10 seconds and not only at startup. – Paul Oct 07 '19 at 12:47

score 0 · Answer 3 · answered Oct 07 '19 at 11:27

0

You configured the same check for readiness and liveness probe - therefore if the liveness check fails, it can be assumed that the readiness fails as well.

answered Oct 07 '19 at 11:27

Thomas

11,272
2
24
40

But shouldn't readiness probe be used only until the pod is up and running? I mean after the pod passes readiness probe, the readiness probe is not being checked anymore? – Maksim Sorokin Oct 07 '19 at 11:45
1

readiness and liveness probes are executed always, if added to the yaml. You can set an interval, but it always checks. – suren Oct 07 '19 at 13:43
The readiness probe is evaluated continuously to determine if an endpoint for the pod should be created as part of a service ("is the application currently ready for production traffic"). Since a temporary problem can occur any time, the readiness check is performed as long as the pod is running. – Thomas Oct 07 '19 at 15:09

score 0 · Answer 4 · answered Oct 07 '19 at 12:55

please provide an implementation function/method at backend, you can make /health named uri, and can write a liveness logic here and readiness can be your choice too.

/health uri, shall be associated with a function implementation which will can return 200 status code if everything goes fine, else it can be made to get failed

Why kubernetes reports "readiness probe failed" along with "liveness probe failed"

4 Answers4

Linked