When the gRPC server is hosted on GKE container, and the service is called by grpcurl after port-forwarding to the target POD, onComplete is not processed and Timeout occurs. And this server, strangely enough, works on the Docker process and on localhost
## Run from sbt console
> sbt server/run
## or Run from Docker image
> sbt clean docker:stage
> docker build -t test-server:latest ./server/target/docker/stage/
> docker run -d -p 8080:8080 test-server:latest
## Successfully requests.
> grpcurl -plaintext -vv -d '{"organizationId": "f"}' 0.0.0.0:8080 com.test.Server/CreateAccount
Resolved method descriptor:
rpc CreateAccount ( .com.test.Request ) returns ( .com.test.Request );
Request metadata to send:
(empty)
Response headers received:
content-type: application/grpc
grpc-accept-encoding: gzip
Estimated response size: 3 bytes
Response contents:
{
"organizationId": "f"
}
Response trailers received:
(empty)
Sent 1 request and received 1 response
And this is the configuration of the app container. The envoy is sidecarred, but it won't affect the app container since the only problem is port-forwarding to it.
- containers
- env:
- name: JAVA_OPTS
value: -Xms1500m -Xmx1500m
- name: TZ
value: Asia/Tokyo
image: asia.gcr.io/my-project/test-server:latest
imagePullPolicy: Always
name: admin
ports:
- containerPort: 8080
name: http
protocol: TCP
resources:
limits:
cpu: "1"
memory: 2000Mi
requests:
cpu: "1"
memory: 2000Mi
securityContext: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-bjhmx
readOnly: true
As a result, there appears to be no header response from the server. However, this could be a possible problem with grpcurl.
> kobectl port-forward server-xxxx-xxxx 8080:8080
> grpcurl -plaintext -vv -d '{"organizationId": "f"}' 0.0.0.0:8080 com.test.Server/CreateAccount
Resolved method descriptor:
rpc CreateAccount ( .com.test.Request ) returns ( .com.test.Request );
Request metadata to send:
(empty)
<-- waiting...
// service implementation
override def CreateAccount(request:Request): Future[Request] = {
println("Received request.")
Future(request)
}
==================
object GrpcInterceptor extends ServerInterceptor with Injector {
override def interceptCall[ReqT, RespT](
serverCall: ServerCall[ReqT, RespT],
headers: Metadata,
next: ServerCallHandler[ReqT, RespT]
): ServerCall.Listener[ReqT] = {
val listener = next.startCall(serverCall, headers)
new ForwardingServerCallListener.SimpleForwardingServerCallListener[ReqT](listener) {
override def onHalfClose(): Unit = {
println("On half close")
super.onHalfClose()
}
override def onComplete(): Unit = {
println("On complete")
super.onComplete()
}
override def onMessage(message: ReqT) {
println("On message")
val value = message.asInstanceOf[GeneratedMessage]
try {
super.onMessage(message)
} catch {
case e: Throwable =>
closeWithException(e, headers)
}
}
private def closeWithException(t: Throwable, requestHeader: Metadata) = {
println("Reciever intercept failed.", t)
throw new StatusRuntimeException(Status.Code.INTERNAL.toStatus.withDescription(t.getMessage).withCause(t))
}
}
}
}
This implementation still does not output the "On complete" log on the GKE container.
On message
On half close
Received request.
However, only the GrpcReflectionApi seems to respond normally.
## On GKE port-forwarding
> grpcurl -plaintext 0.0.0.0:8080 list
com.test.TestServer
grpc.reflection.v1alpha.ServerReflection
I can confirm that it works with the same Docker image, so the only thing I can think of is that it's either a GKE issue or there's some gRPC-specific processing that needs to be done. What other problems can you isolate?
