gRPC HealthCheck reports SERVING even when it is defined as NOT_SERVING

Question

I am trying to create a server with two services and a HealthCheck in each one, so I can check them independently and use reflection to know the methods exposed in each one. I have hardcoded one service as NOT_SERVING to test it, however, for some reason, it is not working. I would appreciate some help.

I have created a short script to reproduce this result.

The proto file definition:

syntax = "proto3";

package test;

service Greeter {
    rpc SayHello (HelloRequest) returns (HelloReply) {}
}

message HelloRequest {
    string name = 1;
}

message HelloReply {
    string message = 1;
}

service Byeer {
    rpc SayBye (ByeRequest) returns (ByeReply) {}
}

message ByeRequest {
    string name = 1;
}

message ByeReply {
    string message = 1;
}

The Python server:

from concurrent import futures
import grpc
from grpc_health.v1 import health, health_pb2, health_pb2_grpc
from grpc_reflection.v1alpha import reflection
import test_pb2_grpc
import test_pb2


class Greeter(test_pb2_grpc.GreeterServicer):

    def SayHello(self, request, context):
        return test_pb2.HelloReply(message='Hello, %s!' % request.name)

    def Check(self, request, context):
        return health_pb2.HealthCheckResponse(status=health_pb2.HealthCheckResponse.SERVING)

    def Watch(self, request, context):
        return health_pb2.HealthCheckResponse(status=health_pb2.HealthCheckResponse.UNIMPLEMENTED)


class Byeer(test_pb2_grpc.ByeerServicer):

    def SayBye(self, request, context):
        return test_pb2.HelloReply(message='Bye, %s!' % request.name)

    def Check(self, request, context):
        return health_pb2.HealthCheckResponse(status=health_pb2.HealthCheckResponse.NOT_SERVING)

    def Watch(self, request, context):
        return health_pb2.HealthCheckResponse(status=health_pb2.HealthCheckResponse.UNIMPLEMENTED)


def run_server():
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=None))
    server.add_insecure_port('[::]:8000')

    test_pb2_grpc.add_GreeterServicer_to_server(Greeter(), server)
    health_pb2_grpc.add_HealthServicer_to_server(Greeter(), server)

    test_pb2_grpc.add_ByeerServicer_to_server(Byeer(), server)
    health_pb2_grpc.add_HealthServicer_to_server(Byeer(), server)

    services = tuple(service.full_name for service in health_pb2.DESCRIPTOR.services_by_name.values())
    services += tuple(service.full_name for service in test_pb2.DESCRIPTOR.services_by_name.values())
    services += (reflection.SERVICE_NAME,)
    reflection.enable_server_reflection(services, server)

    server.start()
    server.wait_for_termination()


if __name__ == "__main__":
    run_server()

When I test the reflection and the server methods, it works:

grpcurl -plaintext localhost:8000 list
grpcurl -plaintext localhost:8000 list test.Byeer
grpcurl -plaintext localhost:8000 list test.Greeter
grpcurl --plaintext -d '{"name": "John Doe"}' localhost:8000 test.Greeter/SayHello
grpcurl --plaintext -d '{"name": "John Doe"}' localhost:8000 test.Byeer/SayBye

The reflections report the proper methods and the server the correct response.

The HealthCheck on the server and the Greeter service work too:

grpcurl --plaintext -d '' localhost:8000 grpc.health.v1.Health/Check
grpcurl --plaintext -d '{"service": "test.Greeter"}' localhost:8000 grpc.health.v1.Health/Check

So, it reports SERVING as expected.

However, the HealthCheck on the Byeer service, and whatever other name I use, also reports SERVING:

grpcurl --plaintext -d '{"service": "test.Byeer"}' localhost:8000 grpc.health.v1.Health/Check
grpcurl --plaintext -d '{"service": "xxx"}' localhost:8000 grpc.health.v1.Health/Check

And I would expect NOT_SERVING since the Check methods for the Byeer has been hardcoded as NOT_SERVING.

Any idea? Thanks in advance.

DazWilkin · Answer 1 · 2021-10-29T00:04:26.180

OK, here's a working example that excludes the reflection (my laziness):

from concurrent import futures
import grpc

from grpc_health.v1.health import HealthServicer
from grpc_health.v1 import health_pb2, health_pb2_grpc

import test_pb2_grpc
import test_pb2


class Greeter(test_pb2_grpc.GreeterServicer):

    def __init__(self,health):
        super().__init__()
        self.health=health

    def SayHello(self, request, context):
        self.health.set(
            "greeter-eater",
            health_pb2.HealthCheckResponse.ServingStatus.Value("SERVING"),
        )
        return test_pb2.HelloReply(message='Hello, %s!' % request.name)

class Byeer(test_pb2_grpc.ByeerServicer):

    def __init__(self,health):
        super().__init__()
        self.health=health

    def SayBye(self, request, context):
        self.health.set(
            "byeer-flyer",
            health_pb2.HealthCheckResponse.ServingStatus.Value("NOT_SERVING"),
        )
        return test_pb2.HelloReply(message='Bye, %s!' % request.name)

def main():
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))

    server.add_insecure_port('[::]:50051')

    # As with any other Servicer implementations, register it
    health = HealthServicer()

    # Provide other services with a way to update health
    greeter = Greeter(health)
    byeer = Byeer(health)

    test_pb2_grpc.add_GreeterServicer_to_server(
        greeter,
        server,
    )
    test_pb2_grpc.add_ByeerServicer_to_server(
        byeer,
        server,
    )
    health_pb2_grpc.add_HealthServicer_to_server(
        health,
        server,
    )

    server.start()
    server.wait_for_termination()


if __name__ == "__main__":
    main()

Without reflection, I needed to grab a copy of health.proto and:

# Invoke Greeter/SayHello
grpcurl \
--plaintext \
-d '{"name": "Freddie"}' \
-import-path ${PWD}/protos \
-proto health.proto \
-proto test.proto \
localhost:50051 test.Greeter/SayHello
{
  "message": "Hello, Freddie!"
}

# Invoke Byeer/SayBye
grpcurl \
--plaintext \
-d '{"name": "Freddie"}' \
-import-path ${PWD}/protos \
-proto health.proto \
-proto test.proto \
localhost:50051 test.Byeer/SayBye
{
  "message": "Bye, Freddie!"
}

# Health/Check Greeter service aka "greeter-eater"
grpcurl \
--plaintext \
-d '{"service":"greeter-eater"}' \
-import-path ${PWD}/protos \
-proto health.proto \
-proto test.proto \
localhost:50051 grpc.health.v1.Health/Check
{
  "status": "SERVING"
}

# Health/Check Byeer service aka "byeer-flyer"
grpcurl \
--plaintext \
-d '{"service":"byeer-flyer"}' \
-import-path ${PWD}/protos \
-proto health.proto \
-proto test.proto \
localhost:50051 grpc.health.v1.Health/Check
{
  "status": "NOT_SERVING"
}

Explanation:

You need only register HealthServicer once per server.
As with any gRPC server, it (HealthServicer), when you invoke its methods (e.g. Check), you need to provide it with some context. In this case, which service.
The service names aren't bound to the proto packages but to an arbitrary string that you define when you set(SERVICE,health_pb2.HealthCheckResponse.ServingStatus)
Because you can register multiple distinct gRPC services (Greeter,Byeer) with a single server, these need to be able to provide health context back to HealthServicer
In order for other gRPC services to update the healthcheck service, we need to share some of the healthcheck service's context with the other services; here I'm using the Servicer itself.
Other services may update their status by using set and providing some (preferably) unique identifier; there's no need for this to match the server's proto package/Service (but that may be a good strategy).
In practice you would not set with static ServingStatus.Value's but would accurately reflect that service's status

Thanks for the help, for the good explanation of why there is a problem in my approach, and to explain how to solve it. Now it works. — Occam's razor, Oct 31 '21 at 23:18

gRPC HealthCheck reports SERVING even when it is defined as NOT_SERVING

1 Answers1