Performance Issue: Async gRPC with Gunicorn + Tornado

Question

Background:

We're trying to migrate our API Gateway from REST to gRPC. The API Gateway will consume by Backend Team with REST, and the communication from API Gateway to microservice will be using gRPC. Our API Gateway build with Tornado Python Framework, Gunicorn, and using tornado.curl_httpclient.CurlAsyncHTTPClient to enable Async / Future for each endpoint. Each endpoint will call to Microservices using Unary RPC and the gRPC stub will return Future.

So before fully migrate to gRPC, we're trying to compare gRPC vs REST performance. Here is the detail u might need to know:

1. We have 3 endpoints to test. /0, /1, and /2 with a single string payload. The payload size are 100KB, 1MB, and 4MB. These message already created when the instance just started, so the endpoint only need to retrieve it.
2. Concurrency = 1, 4, 10 for each endpoint.
3. gRPC Thread pool max workers = 1 and Gunicorn's Worker = 16.
4. We're using APIB for load testing.
5. All the load test is done with GCP VM Instance. The Machine spec is: Intel Broadwell, n1-standard-1 (1 vCPU, 3.75 GB memory), OS: Debian 9
6. The code share the similar structure and same business logic.

Here is the result:

The Conclusion is The higher Concurrency and payload size, the slower gRPC become and eventually slower than REST.

Question:

1. Is gRPC incapable of handling large payload size and large concurrency by using Unary Call compare to REST ?
2. Are there anyway to enable gRPC to become faster than REST ?
3. Are there anything fundamental cause that i missed ?

Here's a few way that i have tried:

1. GZIP Compression from grpcio. The result is it's become slower than before.
2. Using GRPC_ARG_KEEPALIVE_PERMIT_WITHOUT_CALLS and GRPC_ARG_KEEPALIVE_TIMEOUT_MS options on the stub and server config. There's no changes in performance.
3. Change gRPC server max workers to 10000. Result: No Changes in performance.
4. Change Gunicorn Worker to 1. Result: No Changes in Performance.

The way I haven't tried:

1. Using Stream RPC

Any Help is appareciated. Thank you.

Answer 1

Is gRPC incapable of handling large payload size and large concurrency by using Unary Call compare to REST ?

Yes, the gRPC Python bindings are used in production to serve requests as large as several gigabytes at speed.

Are there anyway to enable gRPC to become faster than REST ?

I believe your issue is likely this:

Each endpoint will call to Microservices using Unary RPC and the gRPC stub will return Future.

Each time you use the future API in the Python bindings, a new thread is created to service that request. As you know, Python has a global interpreter lock, so while a process may have many threads, only one of them may access Python objects at any one time. Further, the more threads contending on the GIL, the more slowdowns will happen due to synchronization.

To avoid this, you can either use only the synchronous parts of the gRPC Python API, or you can switch over to the AsyncIO bindings, which were designed to solve exactly this problem.