4

I have a heavy traffic server(more than 800K qps) with go1.7.

From http://urltoserver:debugport/debug/pprof/goroutine?debug=2 I see 8K goroutines out of which almost 1800 are in IO wait for minutes. One of such goroutine stack is as below. 

    goroutine 128328653 [IO wait, 54 minutes]:
    net.runtime_pollWait(0x7f0fcc60c378, 0x72, 0x7cb)
      /usr/local/go/src/runtime/netpoll.go:160 +0x59
    net.(*pollDesc).wait(0xc4231d0a00, 0x72, 0xc42479fa20, 0xc42000c048)
      /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
    net.(*pollDesc).waitRead(0xc4231d0a00, 0x92f200, 0xc42000c048)
      /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
    net.(*netFD).Read(0xc4231d09a0, 0xc423109000, 0x1000, 0x1000, 0x0, 0x92f200, 0xc42000c048)
      /usr/local/go/src/net/fd_unix.go:243 +0x1a1
    net.(*conn).Read(0xc4234282b8, 0xc423109000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
      /usr/local/go/src/net/net.go:173 +0x70
    net/http.(*connReader).Read(0xc420449840, 0xc423109000, 0x1000, 0x1000, 0xc422b38b68, 0x100000000, 0xc421810601)
      /usr/local/go/src/net/http/server.go:586 +0x144
    bufio.(*Reader).fill(0xc422e22360)
      /usr/local/go/src/bufio/bufio.go:97 +0x10c
    bufio.(*Reader).Peek(0xc422e22360, 0x4, 0x7a066c, 0x4, 0x1, 0x0, 0x0)
      /usr/local/go/src/bufio/bufio.go:129 +0x62
    net/http.(*conn).readRequest(0xc422b38b00, 0x931fc0, 0xc424d19440, 0x0, 0x0, 0x0)
      /usr/local/go/src/net/http/server.go:762 +0xdff
    net/http.(*conn).serve(0xc422b38b00, 0x931fc0, 0xc424d19440)
      /usr/local/go/src/net/http/server.go:1532 +0x3d3
    created by net/http.(*Server).Serve
      /usr/local/go/src/net/http/server.go:2293 +0x44d

Anyone have faced this issue ? Any pointer is appreciated.

Vishal Kumar
  • 762
  • 1
  • 7
  • 15
  • What exactly is the issue? Do you want to close active connections? Do you have timeouts set so unresponsive clients don't tie up connections? – JimB Feb 15 '17 at 00:46
  • @JimB how to set connection timeout from server side in golang. I do know for client side. I start my server with http.ListenAndServe – Vishal Kumar Feb 15 '17 at 00:56
  • @VishalKumar Copy the [ListenAndServe code](https://golang.org/src/net/http/server.go#L2349). Modify to set values for [ReadTimeout](https://godoc.org/net/http#Server.ReadTimeout), WriteTimeout, .... – Charlie Tumahai Feb 15 '17 at 01:49

1 Answers1

2

These could easily be clients the initiated a request but never completed it, or slow clients etc.

You should configure the Read/Write timeouts of your server (server.ReadTimeout and server.WriteTimeout respectively):

s := new(http.Server)
// ...
s.ReadTimeout = 5 * time.Second
s.WriteTimeout = 5 * time.Second
// ...
Agis
  • 32,639
  • 3
  • 73
  • 81
  • For completeness changed http.ListenAndServe(":port", r) to s := &http.Server{ Addr: ":port", Handler: r, ReadTimeout: somesec * time.Second, WriteTimeout: sonesec * time.Second, MaxHeaderBytes: 1 << somenumber, } s.ListenAndServe() – Vishal Kumar Feb 15 '17 at 17:48