gRPC streaming call which takes longer than 2 minutes is killed by hardware (routers, etc.) in between client and server

Question

Grpc.Net client:

• a gRpc client sends large amount of data to a gRpc server
• after the gRpc server receives the data from the client, the http2 channel becomes idle (but is open) until the server returns the response to the client
• the gRpc server receives the data and starts processing it. If the data processing takes longer than 2 minutes (which is the default idle timeout for http calls) then the response never reaches the client because the channel is actually disconnected, but the client does not know this because it was shutdown by other hardware in between due to long idle time.

Solution:

• when the channel is created at the gRpc client side, it must have a httpClient set on it
• the httpClient must be instantiated from a socketsHttpHandler with the following properties set (PooledConnectionIdleTimeout, PooledConnectionLifetime, KeepAlivePingPolicy, KeepAlivePingTimeout, KeepAlivePingDelay)

Code snipped:

SocketsHttpHandler socketsHttpHandler = new SocketsHttpHandler()
{
    PooledConnectionIdleTimeout = TimeSpan.FromMinutes(180),
    PooledConnectionLifetime = TimeSpan.FromMinutes(180),
    KeepAlivePingPolicy = HttpKeepAlivePingPolicy.Always,
    KeepAlivePingTimeout = TimeSpan.FromSeconds(90),
    KeepAlivePingDelay = TimeSpan.FromSeconds(90)
};

socketsHttpHandler.SslOptions.RemoteCertificateValidationCallback = (sender, cert, chain, sslPolicyErrors) => { return true; };

HttpClient httpClient = new HttpClient(socketsHttpHandler);
httpClient.Timeout = TimeSpan.FromMinutes(180);

var channel = GrpcChannel.ForAddress(_agentServerURL, new GrpcChannelOptions
{
    Credentials = ChannelCredentials.Create(new SslCredentials(), credentials),
    MaxReceiveMessageSize = null,
    MaxSendMessageSize = null,
    MaxRetryAttempts = null,
    MaxRetryBufferPerCallSize = null,
    MaxRetryBufferSize = null,
    HttpClient = httpClient
});

Answer 1

A workaround is to package your message in an oneof and then send a KeepAlive from a seperate thread every x seconds, for the duration of the calculations.

For example:

message YourData {
   …
}

message KeepAlive {}

message DataStreamPacket {
    oneof data {
       YourData data = 1;
       KeepAlive ka = 2;
    }
}

Then in your code:

stream <-

StartThread() { 
  each 5 seconds:
    Send KeepAlive
}

doCalculations()

StopThread()

SendData()

Answer 2

this is what I needed. I had this problem for months now, but my only solution was to decrease the volume of data.