Golang http server app have many Socket (CLOSE_WAIT)

Question

App can works few days. But in some moment, app has many socket with CLOSE_WAIT state, and cannot receive new client.

Maybe it is some kind of flooding(Example: Sync-flood)?

netstat -ant | grep CLOSE_WAIT | wc

3258 19548 260640

3258 - socket in CLOSE_WAIT state

Update:

Code some handler:

 func GetScore(mongo *mgo.Session, redisConn redis.Conn, renderer handlers.Render) http.Handler {

         mutex := sync.Mutex{}

         return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {

           id := bson.ObjectIdHex(r.FormValue("id"))
           banner := models.Banner{}
           err := mongo.DB("db").C("collection").FindId(id).One(&banner)
           if err != nil {
             log.Panicf("Banner selecting error: %s", err);
           }
           mutex.Lock();
           sports, _ := util.GetSports(redisConn)
           mutex.Unlock();
           sport, _ := sports.FindSport(banner.SportId)
           comp, err := sport.FindCompetition(banner.CompetitionId)
           if err != nil {
               comp, _ = sport.FindCompetition(0);
               log.Println("Competition not found");
           }
           game, err := comp.FindGame(banner.GameId)

           if err != nil {
           game, _ = comp.FindGame(0)
           }
           mutex.Lock();
           scores := util.GetScore(redisConn, game.ID)
           mutex.Unlock();
           game.Score1 = scores[0]
           game.Score2 = scores[1]
           w.Header().Set("Content-Type", "application/json;application/json;charset=utf-8")
           renderer.RenderJson(w, 200, &game)
       }



       func GetScore(redisConn redis.Conn, gameId int) ([]float32) {

         redisKey :=  fmt.Sprintf("game-%d", gameId);
         bBody, err := redis.Bytes(redisConn.Do("GET", redisKey))

         if err != nil || len(bBody) == 0 {
           response, err := http.DefaultClient.Get(fmt.Sprintf("%s%d", GameApi, gameId))

           if err != nil {
             log.Panicf("GetScore error: %s", err)
           }

           bBody, _ = ioutil.ReadAll(response.Body);
           redisConn.Send("SET", redisKey, bBody)
           redisConn.Send("EXPIRE", redisKey, 60 * 5)
           redisConn.Flush()
         }
         events := GameJson{};
         err = json.Unmarshal(bBody, &events)
         if err != nil {
           log.Panicf("GetScore json error: %s", err)
         }

         var event []struct {
           Name string `json:"name"`
           Price float32 `json:"price"`
}
         if len(events.AllEvents.P1XP2) != 0 {
           event = events.AllEvents.P1XP2[0].Events
         } else {
           event = events.AllEvents.Other[0].Events
         }

         return []float32{event[0].Price, event[1].Price}
       }

Answer 1

CLOSE-WAIT means that TCP is waiting for the local application to close the socket, having already received a close from the peer.

Maybe it is some kind of flooding(Example: Sync-flood)?

No. It is a bug in your code. You aren't closing a socket somewhere.

The problem seems to be not in the code.

The problem is in your code.

Code tested several times.

But not for this condition, or under the conditions that produce this problem.

But yesterday I did not have this problem.

So yesterday those condtiions did not occur.

(Code was not changed.)

So the bug has always been there.

Answer 2

Emulation of situation:

When client timeout works. Socket on server will not closed 15 sec. In my case, when app have many request, I have > 10k socket in CLOSE_WAIT STATE.

Emulation of mutex:

time.Sleep(30 * time.Second)

Server:

package main

import "net/http"
import "time";

func main() {

   http.ListenAndServe(":81", http.HandlerFunc(func (w http.ResponseWriter, r *http.Request) {
        println("Sleep")
        time.Sleep(30 * time.Second)
        println("After sleep");
   }));

}

Client:

import java.io.IOException;
import java.net.HttpURLConnection;
import java.net.URL;

public class Main {

    public static void main(String[] args) throws IOException, InterruptedException {
        HttpURLConnection connection = (HttpURLConnection) new URL("http://link").openConnection();
        connection.setRequestMethod("GET");
        connection.setDoOutput(false);
        connection.setDoInput(true);
        connection.setReadTimeout(15000);
        connection.getInputStream().read();
        System.out.println("Close");
        connection.disconnect();
    }
}