grpc_send_timeout doesn't work, Nginx closes GRPC streams unexpectedly

Question

everyone! I have a config for TLS NGINX server, which proxies stream (bidirectional/unidirectional) to my golang GRPC server. I use params in NGINX conf (server context):

grpc_read_timeout 7d;
grpc_send_timeout 7d;

But! My bidirectional streams close after 60s (send data from server frequently, doesn't send any data from client within 60s), as if grpc_send_timeout is set to default value (60s)

But! If I send echo requests from client every 20s it works fine!

I have no idea why grpc_send_timeout doen't work!

nginx.conf:

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log debug;
pid /run/nginx.pid;

include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    resolver 127.0.0.1 valid=10s;
    resolver_timeout 10s;

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    include /etc/nginx/conf.d/*.conf;
}

conf.d/my.service.conf

server {
    listen 443 ssl http2;
    ssl_certificate     my-cert.crt;
    ssl_certificate_key my-key.key;

    access_log "/var/log/nginx/my.service.access.log" main;
    error_log "/var/log/nginx/my.service.error.log" debug;

    grpc_set_header x-real-ip $remote_addr;
    grpc_set_header x-ray-id $request_id;
    grpc_read_timeout 7d;
    grpc_send_timeout 7d; // why it does not work?????

    location /MyGoPackage.MyService {
        grpc_pass grpc://my.service.host:4321;
    }
}

nginx logs:

/ # cat /var/log/nginx/my_host_access.log 
59.932 192.168.176.1 - - [06/May/2021:14:57:30 +0000] "POST /MyGoPackege.MyService/MyStreamEndpoint HTTP/2.0" 200 1860 "-" "grpc-go/1.29.1" "-"

client logs (with GRPC debug logs)

2021-05-06T17:56:30.609+0300    DEBUG   grpc_mobile_client/main.go:39   open connection {"address": "localhost:443"}
INFO: 2021/05/06 17:56:30 parsed scheme: ""
INFO: 2021/05/06 17:56:30 scheme "" not registered, fallback to default scheme
INFO: 2021/05/06 17:56:30 ccResolverWrapper: sending update to cc: {[{localhost:443  <nil> 0 <nil>}] <nil> <nil>}
INFO: 2021/05/06 17:56:30 ClientConn switching balancer to "pick_first"
INFO: 2021/05/06 17:56:30 Channel switches to new LB policy "pick_first"
INFO: 2021/05/06 17:56:30 Subchannel Connectivity change to CONNECTING
INFO: 2021/05/06 17:56:30 Subchannel picks a new address "localhost:443" to connect
INFO: 2021/05/06 17:56:30 pickfirstBalancer: HandleSubConnStateChange: 0xc0004b2d60, {CONNECTING <nil>}
INFO: 2021/05/06 17:56:30 Channel Connectivity change to CONNECTING
INFO: 2021/05/06 17:56:30 Subchannel Connectivity change to READY
INFO: 2021/05/06 17:56:30 pickfirstBalancer: HandleSubConnStateChange: 0xc0004b2d60, {READY <nil>}
INFO: 2021/05/06 17:56:30 Channel Connectivity change to READY
2021-05-06T17:56:30.628+0300    DEBUG  main.go:54   open stream      {"address": localhost:443"}
2021-05-06T17:56:30.974+0300    INFO   main.go:81   new msg from server     {"msg": "hello world"}

// some logs within a 60s

2021-05-06T17:57:30.567+0300    FATAL  main.go:79   receive new msg from stream     {"error": "rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR"}

server logs (only this one at the moment of connect closing, GRPC debug log):

INFO: 2021/05/06 17:57:30 transport: loopyWriter.run returning. connection error: desc = "transport is closing"

I have faced the same issue in my .net core GRPC-backend. Nginx closes the connection after 60s. All smth_timeout properties don't work. It seems a problem with nginx. — Oleg Varlamov, Jun 01 '21 at 12:14

score 4 · Answer 1 · answered Jun 02 '21 at 12:52

4

client_header_timeout 7d;
client_body_timeout 7d;

adding this params to nginx conf solved the problem

answered Jun 02 '21 at 12:52

Владимир Савостин

464
4
9

1

It didn't work for me putting it within the "http" bracket, but within the "server" bracket your solution worked! – Blobfisher Oct 15 '21 at 10:45

grpc_send_timeout doesn't work, Nginx closes GRPC streams unexpectedly

1 Answers1