Elegant way to handle goroutine in IO wait state in GRPC

Question

Our server (grpc-gateway + grpc) is running on K8S with go 1.13, and terminate stack information

    Last State:    Terminated
      Reason:      Error
      Message:     o.(*Reader).fill(0xc002ec78c0)
                   /Users/local/go/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).Peek(0xc002ec78c0, 0x4, 0x0, 0x0, 0x0, 0x0, 0xc002e79ad0)
  /Users/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*conn).readRequest(0xc00245a000, 0x1ef4800, 0xc000be7780, 0x0, 0x0, 0x0)
  /Users/local/go/src/net/http/server.go:962 +0xb3b
net/http.(*conn).serve(0xc00245a000, 0x1ef4800, 0xc000be7780)
  /Users/local/go/src/net/http/server.go:1817 +0x6d4
created by net/http.(*Server).Serve
  /Users/local/go/src/net/http/server.go:2928 +0x384

goroutine 8724981 [IO wait]:
internal/poll.runtime_pollWait(0x7f5d3a8f84e8, 0x72, 0xffffffffffffffff)
  /Users/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc000155d98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
  /Users/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
  /Users/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000155d80, 0xc00138b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
  /Users/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc000155d80, 0xc00138b000, 0x1000, 0x1000, 0xc00142b9e8, 0x4ce13d, 0xc000155d80)
  /Users/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0000dd690, 0xc00138b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
  /Users/local/go/src/net/net.go:184 +0x68
net/http.(*connReader).Read(0xc001e9b050, 0xc00138b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
  /Users/local/go/src/net/http/server.go:785 +0xf4
bufio.(*Reader).fill(0xc0021fe060)
  /Users/local/go/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).Peek(0xc0021fe060, 0x4, 0x0, 0x0, 0x0, 0x0, 0xc00142bad0)
  /Users/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*conn).readRequest(0xc0008f43c0, 0x1ef4800, 0xc000503640, 0x0, 0x0, 0x0)
  /Users/local/go/src/net/http/server.go:962 +0xb3b
net/http.(*conn).serve(0xc0008f43c0, 0x1ef4800, 0xc000503640)
  /Users/local/go/src/net/http/server.go:1817 +0x6d4
created by net/http.(*Server).Serve
  /Users/local/go/src/net/http/server.go:2928 +0x384

      Exit Code:    2

Per this question, one possible solution is

>  s := new(http.Server)
>  // ...
>  s.ReadTimeout = 5 * time.Second
>  s.WriteTimeout = 5 * time.Second
>  // ...

However, we failed to find the ReadTimeout from grpc.NewServer, is there anything are we missing? or how to handle goroutine in IO wait state in GRPC more elegantly?

grpc version is v1.21.1

score 1 · Answer 1 · answered Jun 15 '21 at 15:12

gprc.NewServer allows for zero or more ServerOption during creation:

func NewServer(opt ...ServerOption) *Server

while there does not appear to be a ReadTimeout or WriteTimeout equivalent to http.Server you could try keepalive.ServerParameters:

type ServerParameters struct {
    MaxConnectionIdle     time.Duration // The current default value is infinity.
    MaxConnectionAge      time.Duration // The current default value is infinity.
    MaxConnectionAgeGrace time.Duration // The current default value is infinity.
    Time                  time.Duration // The current default value is 2 hours.
    Timeout               time.Duration // The current default value is 20 seconds.
}

(full docs keepalive.ServerParameters)

and tune say keepalive.ServerParameters.Time to something lower than 2 hours:

srv := grpc.NewServer(
    keepalive.ServerParameters{Time:5*time.Minute},
)

This will lower connection reuse - but will also free up client connections that are long since dead.

score -1 · Answer 2 · answered Jun 15 '21 at 11:34

-1

You use Context with Timeout.

https://golang.org/pkg/context/

answered Jun 15 '21 at 11:34

Moo

87
6

1

That's a client-side solution. The OP is asking about the server-side. – colm.anseo Jun 15 '21 at 12:17
What? context.Context is server-side! http.request.context() – Moo Jun 15 '21 at 12:26
https://ieftimov.com/post/make-resilient-golang-net-http-servers-using-timeouts-deadlines-context-cancellation/ – Moo Jun 15 '21 at 12:30
1

That is the server-side request context once a request is being handled. It can be used to detect cancelation of a live request. What if the client has a connection, but hasn't sent a request yet? – colm.anseo Jun 15 '21 at 12:43

Elegant way to handle goroutine in IO wait state in GRPC

2 Answers2