0

Stack Overflow message

I am making a project on Google Cloud Run. It is a microservice architecture with different Go services communicating with each other via gRPC. The project has HTTP endpoints exposed with ESPv2 which handles the transcoding of HTTP to gRPC.

When I send gRPC messages to the services directly, they work correctly. I’m able to do all CRUD functions successfully. When I send an HTTP request to the service URL, I get an error:

{
    "code": 12,
    "message": "unknown service <package>.<MyAPIGateway>"
}

This is my simplified api_config.yaml in Api Gateway service

type: google.api.Service
config_version: 3
name: <service url>.a.run.app
title: My Microservices
apis:
  - name: package.MyAPIGateway
usage:
  rules:
  # Methods can be called without an API Key.
  - selector: package.MyAPIGateway.*
    allow_unregistered_calls: true
backend:
  rules:
    - selector: package.MyAPIGateway.GetUser
      address: grpcs://user-service.a.run.app

#
# HTTP rules define translation from HTTP/REST/JSON to gRPC. With these rules
# HTTP/REST/JSON clients will be able to call the service.
#
http:
  rules:
  - selector: package.MyAPIGateway.GetUser
    get: /user

In User service, I register a gRPC service

// Create gRPC server
    grpcServer := grpc.NewServer()

    // Register service with gRPC server
    userpb.RegisterUserServiceServer(grpcServer, userService)

However, the API Gateway has no Go code, so it is never registered, which I believe causes the error. Does API Gateway need to have code to handle requests? The intent was for API Gateway to just configure the project and the User Service would handle the business logic.

I attempted to update the api_config.yaml to assign the rules directly to the User Service

backend:
  rules:
    - selector: userpb.UserService.GetUser
      address: grpcs://user-service.a.run.app

#
# HTTP rules define translation from HTTP/REST/JSON to gRPC. With these rules
# HTTP/REST/JSON clients will be able to call the service.
#
http:
  rules:
  - selector: userpb.UserService.GetUser
    get: /user

But this resulted in the error:

{
    "code": 503,
    "message": "upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection failure, transport failure reason: delayed connect error: 111"
}

Any guidance for resolving this communication error would be much appreciated. Thanks!

Here is the log entry for the 501 request

{
  "insertId": "<ID>",
  "httpRequest": {
    "requestMethod": "GET",
    "requestUrl": "https://<URL>.a.run.app/user",
    "requestSize": "649",
    "status": 501,
    "responseSize": "1273",
    "userAgent": "PostmanRuntime/7.32.2",
    "remoteIp": "<IP>",
    "serverIp": "<IP>",
    "latency": "3.539549s",
    "protocol": "H2C"
  },
  "resource": {
    "type": "cloud_run_revision",
    "labels": {
      "project_id": "<project ID>",
      "location": "us-central1",
      "revision_name": "<Project Revision>",
      "configuration_name": "<project>",
      "service_name": "<project>"
    }
  },
  "timestamp": "2023-06-03T18:04:22.158774Z",
  "severity": "ERROR",
  "labels": {
    "instanceId": "<instance id>"
  },
  "logName": "projects/<project ID>/logs/run.googleapis.com%2Frequests",
  "trace": "projects/<project ID>/traces/ff037d1ab301a770ab20f3d74eb68d5f",
  "receiveTimestamp": "2023-06-03T18:04:25.862194414Z",
  "spanId": "<span id>",
  "traceSampled": true
}
MikeBird
  • 386
  • 1
  • 5
  • 17
  • Please include the logs from the Cloud Endpoints proxy (when making the HTTP GET request). – DazWilkin Jun 03 '23 at 16:08
  • The Cloud Endpoints (ESPv2) proxy is configured entirely by the service configuration (there is no user code to the proxy). What you show (`api_config.yaml`) looks to be sufficient and, after deploying it (`gcloud endpoints service deploy`), you should have a Cloud Run service that represents it. – DazWilkin Jun 03 '23 at 16:09
  • When you say that you can send messages to the services directly, do you mean to the Cloud Run service that implements your gRPC code ( `userService`)? Or are you (as you should try) sending gRPC messages through the proxy to ensure this path works correctly? – DazWilkin Jun 03 '23 at 16:10
  • I'm familiar with using Cloud Endpoints to provide a gRPC proxy to gRPC services but haven't used the JSON|gRPC transcoding. Unless `package.MyAPIGateway.GetUser` takes an `Empty` request message, you'll need to configure the HTTP `get` to provide the parameters to the service's method. – DazWilkin Jun 03 '23 at 16:12
  • **It likely makes no difference in terms of resolving the issue** but, Google recommends that you annotate the Protobuf source with the HTTP mappings rather than use the service configuration ([Where to configure transcoding](https://cloud.google.com/endpoints/docs/grpc/transcoding#where_to_configure_transcoding)) – DazWilkin Jun 03 '23 at 16:15
  • The `503` suggests to me that you've deployed everything correctly but that you've not mapped the HTTP `GET` for `GetUser` correctly. I suspect you could e.g. `gRPCurl` it correctly through the proxy (if you provide the request message). – DazWilkin Jun 03 '23 at 16:17
  • I am not getting much information from the logs but I will update the post with the entry for the 501. Please let me know if there's anything I should search for in the logs that will help provide more detail – MikeBird Jun 03 '23 at 18:08
  • I send gRPC messages directly to the User Service via Postman. What are you referring to when you say "the proxy"? – MikeBird Jun 03 '23 at 18:12
  • Cloud Endpoints (ESPv2). You've configured it to accept JSON/HTTP but it will also accept pure gRPC requests. So, you should be able to route your User services requests through Cloud Endpoints to confirm that gRPC routing is working. Then, if it is, you can focus on ensuring the JSON/HTTP mapping is correct (I think it's not). – DazWilkin Jun 03 '23 at 18:58
  • Ok. HTTP 503 (Service Unavailable) and 501 (Method Unimplemented) are different errors so I'm slightly confused. However, 501 suggests (!) that the proxy is attempting to call your backend gRPC service and being told that the method isn't implemented. You reference this backend service method as `package.MyAPIGateway.GetUser` which I assume is correct. When the proxy logs the 501 at `2023-06-03T18:04:22.158774Z`, there should be a corresponding log entry (~same time) in the backend service's request log showing the request being made against the gRPC service. – DazWilkin Jun 03 '23 at 19:14
  • The proxy logs quite extensively. You're showing the requests log (`/run.googleapis.co/requests`) but there's also a specific log for the proxy (`endpoints_log`) and there are stdout|stderr logs. These will provide extensive information. – DazWilkin Jun 03 '23 at 19:15
  • I'm headed out for the afternoon but will check in on this thread later|tomorrow. Try sending a message to `GetUser` through the proxy's endpoint (not the backend service's endpoint). What is the request message for `GetUser`? The HTTP annotation (`get: /user`) suggests that it takes no parameters which would require it be defined as `rpc GetUser (google.protobuf.Empty) returns (...)`. Is that true? If it's not `Empty`, then you need to define the parameters|querystring. – DazWilkin Jun 03 '23 at 19:22
  • I think you're right that the mapping is incorrect. In User-Service, user.proto: `service UserService { rpc GetUser(GetUserRequest) returns (User) {} ... message GetUserRequest { string id = 1; }` – MikeBird Jun 04 '23 at 20:52
  • I can send a gRPC message to `user-service-...-uc.a.run.app` and it is successful. If I send a gRPC message to `api-gateway-...-uc.a.run.app` it responds with a `12 unimplemented` – MikeBird Jun 04 '23 at 20:54
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/253948/discussion-between-mikebird-and-dazwilkin). – MikeBird Jun 04 '23 at 20:55
  • Clearly, this suggests that the proxy is incorrectly configured. The service config **looks** fine. Is your fully-qualified method name `userpb.UserService.GetUser`? Is that the method that you're invoking through Postman? I know Stack overflow is pestering you to move the discussion to chat but I don't have much more time to help with this issue. – DazWilkin Jun 05 '23 at 15:17
  • Yes to both questions. And not a worry, I appreciate the help you've provided. – MikeBird Jun 07 '23 at 22:18

0 Answers0