exec probe in GKE

Question

I'm trying to use exec probes for readiness and liveness in GKE. This is because it is part of Kubernetes' recommended way to do health checks on gRPC back ends. However when I put the exec probe config into my deployment yaml and apply it, it doesn't take effect in GCP. This is my container yaml:

  - name: rev79-uac-sandbox
    image: gcr.io/rev79-232812/uac:latest
    imagePullPolicy: Always
    ports:
    - containerPort: 3011
    readinessProbe:
      exec:
        command: ["bin/grpc_health_probe", "-addr=:3011"]
      initialDelaySeconds: 5
    livenessProbe:
      exec:
        command: ["bin/grpc_health_probe", "-addr=:3011"]
      initialDelaySeconds: 10

But still the health checks fail and when I look at the health check configuration in the GCP console I see a plain HTTP health check directed at '/'

When I edit a health check in GCP console there doesn't seem to be any way to choose an exec type. Also I can't see any mention of liveness checks as contrasted to readiness checks even though these are separate Kubernetes things.

Does Google cloud support using exec for health checks? If so, how do I do it? If not, how can I health check a gRPC server?

Have you taken a look at https://kubernetes.io/blog/2018/10/01/health-checking-grpc-servers-on-kubernetes/ ? — Richard Belleville, May 01 '19 at 18:04
@Richard Belleville, yes. That's where I learnt about this from. But it doesn't work in GKE — Toby 1 Kenobi, May 01 '19 at 22:17
@RichardBelleville the server has to implement the grpc probe protocol — unludo, May 15 '19 at 16:16
@Toby1Kenobi..these probes working fine for me 1) downloaded grpc health go utility in my docker image 2) implemented health proto in my grpc server 3) added exec probes in yaml, it shows SERVING status when I do kubect exec -it -- /bin/bash — Abhay, Oct 07 '19 at 21:01
@Abhay yes, that much works fine, but how to get the load balancer health checks to execute that in GKE? — Toby 1 Kenobi, Oct 08 '19 at 18:14
Why you're expecting load balancer's health check to use this grpc exec probe? The load balancer will have to do its own health check by querying some end point. Kubernetes health check is done by kubelet and uses the pod IP load balancer IP or ingress IP. Am I missing something here? — Abhay, Oct 08 '19 at 19:20
@Abhay you are right. I was getting the Kubernetes probes confused with load balancer health checks, but they are different things. — Toby 1 Kenobi, Apr 26 '21 at 12:42

score 1 · Answer 1 · answered Apr 25 '19 at 15:30

TCP probes are useful when we are using gRPC Services rather than using HTTP probes.

    - containerPort: 3011
    readinessProbe:
      tcpSocket:
        port: 3011
      initialDelaySeconds: 5
      periodSeconds: 10
    livenessProbe:
      tcpSocket:
        port: 3011
      initialDelaySeconds: 15
      periodSeconds: 20

the kubelet will attempt to open a socket to your container on the specified port. If it can establish a connection, the container is considered healthy, if it can’t it is considered a failure define-a-tcp-liveness-probe

Thanks for this. It seems I don't need to use an exec probe after all. However when I set these tcp probes the changes don't take effect, even after waiting 10 minutes. When I try to change the health checks to be TCP through the Google Cloud Console, they get changed back to HTTP after a few minutes. — Toby 1 Kenobi, Apr 25 '19 at 23:38

score 0 · Answer 2 · answered Apr 25 '19 at 15:47

0

Exec probes work in GKE just the same way they work everywhere. You can view liveness probe result in "kubectl describe pod". Or you can simply log in into pod, execute command and see its return code.

answered Apr 25 '19 at 15:47

Vasili Angapov

8,061
15
31

score 0 · Answer 3 · answered May 15 '19 at 16:18

0

The server has to implement the grpc probe protocol as indicated here as indicated in this article

answered May 15 '19 at 16:18

unludo

4,912
7
47
71

score -1 · Answer 4 · answered Apr 26 '19 at 01:40

Both answers from Vasily Angapov and Suresh Vishnoi should in theory work, however in practice they don't (at least in my practice).

So my solution was to start another server on my backend container - an HTTP server that simply has the job of executing the health check whenever it gets a request and returning a 200 status if it passes and a 503 if it fails.

I also had to open a second port on my container for that server to listen on.

exec probe in GKE

4 Answers4