0

I am using gorilla mux to create a golang server to support a simple health GET endpoint. The endpoint responds with a status of ok whenver the server is up. I see a lot of connections (over 400) in CLOSE_WAIT state on one system. This does not happen on other systems with the same code. Output of netstat (9003 is my server port):

tcp      164      0 ::1:9003                    ::1:60702                   CLOSE_WAIT  -
tcp      164      0 ::1:9003                    ::1:44472                   CLOSE_WAIT  -
tcp      164      0 ::1:9003                    ::1:31504                   CLOSE_WAIT  -

This seems to imply that I have a connection I need to close. Most of the questions I read online seem to suggest that open connections pertain to the client not issuing a response.body.close() after a GET. As per https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/, I could add read/write timeouts on server side but I would like to understand the root cause of CLOSE_WAITS before adding the improvements.

Am I missing any close on the server side?

My code is below:

import  "github.com/gorilla/mux"
...

func (server *Srvr) healthHandler(w http.ResponseWriter, r *http.Request) {
    resp := map[string]string{"status": "ok"}
    respJSON, err := json.Marshal(resp)
    if err != nil {
        w.WriteHeader(http.StatusInternalServerError)
        fmt.Fprintf(w, "Error creating JSON response %s", err)
        return
    }
    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(http.StatusOK)
    w.Write(respJSON)
}


// Load initializes the servers
func Load(port string) *Srvr {
    srvrPort := ":" + port
    log.Infof("Will listen on port %s", srvrPort)
    serverMux := mux.NewRouter()
    srvr := &Srvr{Port: port, Srv: &http.Server{Addr: srvrPort, Handler: serverMux}}
    serverMux.HandleFunc("/api/v1.0/health", srvr.healthHandler).Methods("GET")
    return srvr
}

// Run starts the server
func (server *Srvr) Run() {
    log.Info("Starting the server")

    // Starting a server this way to allow for shutdown.
    // https://stackoverflow.com/questions/39320025/how-to-stop-http-listenandserve
    err := server.Srv.ListenAndServe()
    if err != http.ErrServerClosed {
        log.Fatalf("ListenAndServe(): %s", err)
    }
}
// Main resides outside the server package
func main() {

    srvr := server.Load("9003")
    // Now that all setup is done successfully, lets start the server
    go srvr.Run()
    // An unrelated forever loop executes below for different business logic
for {
        glog.Info("Evaluation iteration begins now")
        ...

        time.Sleep(time.Duration(evalFreq) * time.Minute)
    }

}
peaxol
  • 497
  • 4
  • 13
  • Server code looks ok, except that in main, when you do `go srvr.Run()`, that should start the server in a separate goroutine and then terminate the program. Are you sure this is really the code you're running? – Burak Serdar Feb 08 '20 at 00:13
  • Thanks for pointing it out. I have a forever loop for some other business logic after the go run. Added it in the question's code now. – peaxol Feb 08 '20 at 00:23
  • This isn't go-specific, CLOSE_WAIT is part of normal TCP operation, it doesn't mean you "haven't closed" a connection; it's just part of what happens when a TCP socket shuts down. See: https://stackoverflow.com/questions/15912370/how-do-i-remove-a-close-wait-socket-connection and https://webmasters.stackexchange.com/questions/22946/what-sense-can-be-made-out-of-time-wait-close-wait-established – BadZen Feb 08 '20 at 00:23
  • I have over 400 connections in CLOSE_WAIT leading me to believe something is off here. Other systems where this code is running do not see that issue. – peaxol Feb 08 '20 at 00:25
  • @peaxol for how long do they stay in the `CLOSE_WAIT` state? – zerkms Feb 08 '20 at 00:36
  • How do I check how long a connection has been in CLOSE_WAIT @zerkms? I can only see that the number of connections in CLOSE_WAIT has stayed high over the last day or so since I have been monitoring the system. It is currently at 353, I can keep a tab of how it changes periodically. – peaxol Feb 08 '20 at 00:45
  • https://superuser.com/questions/565991/how-to-determine-the-socket-connection-up-time-on-linux https://serverfault.com/questions/296853/determining-at-what-time-a-currently-open-tcp-connection-was-created – zerkms Feb 08 '20 at 00:46
  • @zerkms - Cant find out. Tried a couple of ways. netstat -o gives the time as `off (0.00/0/0)` lsof doesnt give any information on the close_wait connections – peaxol Feb 08 '20 at 01:16

0 Answers0