How to close a SOCKET when using IOCP?

Question

I have a server application that uses IOCP. I want to know what is the proper way to close a SOCKET.

If I simply call closesocket() (for a SOCKET with a handle of for example 12345), and this SOCKET has pending IO operations (for example: a pending WSARecv() request), then the following scenario could happen:

• I call closesocket() which will destroy the SOCKET.

• I accept another SOCKET with the same handle of 12345.

• I deque the pending WSARecv() completion packet for the SOCKET with the handle of 12345. Now I would assume that this completion packet is for the current SOCKET with the handle of 12345, but in fact it is for the SOCKET that was previously closed (this is he main problem with this approach).

So this is obviously a bad approach.

The second approach that seems correct is the following:

• I associate a struct instance with each SOCKET. The struct has the following members: an int called number_of_pending_IO_operations, and a boolean called Is_SOCKET_being_closed.

• When I issue an IO operation for the SOCKET (for example: a WSASend() request), I increment number_of_pending_IO_operations by 1, and when I deque a completion packet for the SOCKET, I decrement number_of_pending_IO_operations by 1.

• Now, when I want to close the SOCKET, I don't simply call closesocket(), but rather I call CancelIOEx() to cancel all pending IO operations for the SOCKET, and I also set Is_SOCKET_being_closed to true.

• When I am about to issue another IO operation (for example: a WSASend() request), I would check the value of Is_SOCKET_being_closed, and if it is true, I would not issue the IO operation.

• Now I simply wait for all of the completion packets to be dequeued, and when number_of_pending_IO_operations reaches 0 and Is_SOCKET_being_closed is set to true, I call closesocket().

Of course I would have race conditions, so I would use critical sections.

Is the second approach a correct way to close a SOCKET, or is there a better way?

Answer 1

I had a similar issue in my client-server application. I think I have no races, no rundown protection, and no critical sections. There are some tradeoffs though.

• only one WSARecv() at a time. Multiple buffers -- maybe, but only one WSARecv(). WSARecv completes (possibly, inline), packet pops from completion port, I quickly check what I got and issue another WSARecv()
• I have a flag (actually, a counter that can only go up) for Is_SOCKET_being_closed. Counter because I may simultaneously decide to kill the socket in two different places.
• after setting Is_SOCKET_being_closed, I call shutdown() followed by CancelIoEx() which will break the previously issued WSARecv().
• there is a race between CancelIoEx and completion of WSARecv() -- I eliminate it with double checking Is_SOCKET_being_closed immediately before issuing WSARecv(). Even if race occurs, WSARecv() is doomed to fail because shutdown().
• the structure is refcounted, where one ref is held by the receive state machine, and one ref is held for every legal owner of the ptr (as in, potential sender). If you have a legal ref, you can make a copy which you can hand off to someone else (it differs here from rundown references as their AddRef() may fail). I will only closesocket() when that ref drains to zero, because I cannot guarantee that there is no thread that has passed all the checks and about to issue a WSARecv/WSASend.