0

I needed help in setting up GRPC for an ML model deployed on GCP. I have deployed the model and want to serve it using GRPC but not able to find any documentation on how to do that. Does anyone have any info on the below:-

  1. How to activate GRPC for TF serving model deployed at ml.googleapis.com?
  2. What are the steps that are needed for the model to be served using GRPC with authorization?

Currently there is sufficient documentation for serving with JSON requests but these requests are becoming very big and I need an alternative for reducing size and thats why needed to find more doc on it. right now I am getting error

InactiveRpcError of RPC that terminated with:
    status = StatusCode.UNKNOWN
    details = "Stream removed"

let me know if you would need more info.

this the code i am using

from google import auth as google_auth
from google.auth.transport import requests as google_auth_transport_requests
from google.auth.transport import grpc as google_auth_transport_grpc
from tensorflow_serving.apis import predict_pb2

def _make_grpc_request(examples):
    scoped_credentials, _ = google_auth.default(scopes=('https://www.googleapis.com/auth/cloud-platform',))
    request = google_auth_transport_requests.Request()
    channel = google_auth_transport_grpc.secure_authorized_channel(
    scoped_credentials, request, 'ml.googleapis.com:443')

    """Builds and sends request to TensorFlow model server."""
    request = predict_pb2.PredictRequest()
    request.model_spec.name = 'nudity2'                           
    request.model_spec.signature_name = 'serving_default'
    request.inputs["input"].CopyFrom(
        tf.make_tensor_proto(examples))
    p_s=PredictionServiceStub(channel)
    response = p_s.Predict(request, 1000)
    response
    return response

    return _make_grpc_request
user9063834
  • 85
  • 1
  • 8
  • Just an update on checking in logs of GCP I found that I am getting textPayload: "ERROR:root:The request body needs to be utf-8 encoded." ,I am using a numpy and base64.b64encode(image).decode('utf-8') .. any ideas ? – user9063834 Oct 04 '20 at 16:02
  • gRPC is not supported at this time. What is the format of your input? Are you using float32 tensors for images? If so, consider using png- or jpg-compressed images instead. Will that work? – rhaertel80 Oct 09 '20 at 00:45

0 Answers0