PATCH API don't work on Google Cloud Run instance

Question

I have cloud run services hosting GO OSB application implementing gRpc but exposing the http REST api's via grpc-gateway which uses cloud sql (mysql) as a DB. All the CRUD API's are responding fine except the PATCH one.

It's throwing the below error with http response code 503:

{
  "textPayload": "The request failed because either the HTTP response was malformed or connection to the instance had an error.",
  "insertId": "6141e984000c63529e7b7afd",
  "httpRequest": {
    "requestMethod": "PATCH",
    "requestUrl": "https://********-********-mr336-qv7hk7cx3a-uc.a.run.app/v2/service_instances/237e80fd-b22e-4df0-b9ed-23c91a4d7f51",
    "requestSize": "1102",
    "status": 503,
    "responseSize": "976",
    "userAgent": "PostmanRuntime/7.28.4",
    "remoteIp": "********",
    "serverIp": "********",
    "latency": "0.410343680s",
    "protocol": "HTTP/1.1"
  },
  "resource": {
    "type": "cloud_run_revision",
    "labels": {
      "location": "us-central1",
      "revision_name": "********-********-mr336-00001-hop",
      "project_id": "********-********-l-app-us-01",
      "configuration_name": "********-********-mr336",
      "service_name": "********-********-mr336"
    }
  },
  "timestamp": "2021-09-15T12:39:32.811858Z",
  "severity": "ERROR",
  "labels": {
    "instanceId": "00bf4bf02dff6d5f53cff1f1828cafbca265606a996eddff5cb44e3fff674efb77ca51eca7087fb8b8e7acba227b2a3e3e913bdfcc0a487640a2e028"
  },
  "logName": "projects/********/logs/run.googleapis.com%2Frequests",
  "trace": "projects/********/traces/e29e5add9452d171e9eebd26817bb667",
  "receiveTimestamp": "2021-09-15T12:39:32.817171397Z"
}

Points to Note :

After every patch request I can see the instance start-up logs, i.e. after the above mentioned logs every time I can see the container entrypoint (server) startup logs (like cold-start).

As soon as server startup is complete, it again throws the same above error in logs.

Important point to note is that I also can't see any logs from my application which suggests PATCH api request is not reaching the container instances running behind the cloud run services.

Also my active instances after cold start goes to ideal and then scales down to 0 with in a 1 min. after the last request hit, but it doesn't seems to create issue for other APIs. This is how it is supposed to work but can't find any lead on what's the issue with PATCH.

this sounds very similar to this: https://stackoverflow.com/questions/58493512/cant-get-http-patch-to-work-on-google-cloud-run-instance — Caner, Sep 15 '21 at 13:43
@Caner I looked into that, but in my case request is not even reaching the container instance and DB transactions comes after that. So, I think that will be out of scope here. Although I have tested it on local setup with a application container and a mysql container, its working fine. — ak89224, Sep 15 '21 at 14:44
@ak89224 Can you please share your dockerfile with us. I see that reqeusts are arriving even before the container is fully operational. Also your instances were scaling back to 0 constantly. — Rajeev Tirumalasetty, Sep 16 '21 at 10:31
@RajeevTirumalasetty I fixed the issue. I'll provide the answer. Although in dockerfile i use entrypoint for server startup. — ak89224, Sep 16 '21 at 10:51

score 1 · Accepted Answer · answered Sep 16 '21 at 11:08

This is fixed now !

Facing the due to handling of multiple protocols on the same port and one of the protocol matchers was causing the issue with PATCH API by returning Empty reply from server. So, done changes to the matchers and it worked.

RCA : Cmux HTTP1Fast Matcher only matches the methods in the HTTP request. This matcher is very optimistic: if it returns true, it does not mean that the request is a valid HTTP response. A correct but slower HTTP1 matcher, used "HTTP1" instead which scan the whole request up-to 4096 bytes but its bit slow.

PATCH API don't work on Google Cloud Run instance

1 Answers1