0

While busy working with Windows Sockets in overlapped mode and using Completion routines (so no IOCP) for feedback I found the following curious case:

  1. Open a server socket using listen and AcceptEx.
  2. Connect a Client socket on said port using ConnectEx

We now have (at least) 3 sockets: 1 listing socket, a client connected socket and a server connected socket.

after transferring some data we close both the server and client connected sockets with shutdown. After this step both sockets are closed with closesocket.

Currently: just to be sure we have no pending completion routine I issue the following (pseudocode):

while SleepEx( 0, TRUE ) == WAIT_IO_COMPLETION do ;

I thought now it would be save to free the memory of the OVERLAPPED structures used by WSARecv and WSASend.

After this moment when the thread becomes in an alertable state again another completion routine callback is done for the server connected socket with an error 10053 but using the OVERLAPPED structure we just freed. This is use of memory after free.

Question:

When can you be sure no completion callbacks are issued anymore for a socket using overlapped IO using completion routines?

Ritsaert Hornstra
  • 5,013
  • 1
  • 33
  • 51
  • @RbMm: There's no IOCP here. – Ben Voigt Jul 26 '18 at 14:56
  • @BenVoigt - but how about `AcceptEx` and `ConnectEx` ? only way use it in asynchronous mode - bind socket for iocp. unclear what is OP doing – RbMm Jul 26 '18 at 14:58
  • @RbMm: You can do asynchronous I/O just fine without completion ports. – Ben Voigt Jul 26 '18 at 15:00
  • @BenVoigt - depend from api. `AcceptEx` and `ConnectEx` alow asynchronous only with iocp. no APC option for this api, no callback here as parameter. are not ? or OP use synchronous `AcceptEx` and `ConnectEx` ? – RbMm Jul 26 '18 at 15:02
  • @RbMm: No callback but there's an event handle in the overlapped structure. And the actual operation causing him problems is the `WSARecv` which does have a callback parameter. – Ben Voigt Jul 26 '18 at 15:28
  • @BenVoigt - yes, but use events very uncomfortable - need have unique event for every io request (so usually create/close events), then - when event signaled - need somehow manually get from event pointer to overlapped (need use some map for this). because this i very not like events completion. however i first wrong read question, and your answer is correct – RbMm Jul 26 '18 at 15:36
  • @RbMm: Only *concurrent* I/O requests need distinct events. For example if you're just streaming data from a serial port (so that each time a read completes, you issue a new overlapped read) then the new read can reuse the same OVERLAPPED and hEvent (and if you like, data buffer) as the completed one. Also, the return value from `(Msg)WaitForMultipleObjects(Ex)` is useful for finding which operation completed. – Ben Voigt Jul 26 '18 at 16:05
  • @RbMm: So yeah, for a web server with a variable number of simultaneous clients, completion ports are definitely the way to go. But for a desktop app with exactly one connection to a server, exactly one connection to a log file, exactly one connection to a local serial peripheral... overlapped I/O with fixed indexes works very smoothly. – Ben Voigt Jul 26 '18 at 16:08

1 Answers1

2

You need to wait for the I/O completion (closing the socket will cancel outstanding requests and you will get a completion callback).

The OS has ownership of the OVERLAPPED structure and associated buffer until you synchronize on event completion (by waiting for the hEvent or receiving an APC). You cannot do anything with the buffer until you receive this callback, and you definitely must not free it. Wait for the OS to tell you it is no longer needed.

Note that cancellations don't necessarily cause completion immediately, because the driver may be synchronizing with hardware requests and only mark the IRP complete when the hardware state changes. (This would be necessary if DMA is in use but might be done for other operations just for consistency) So the SleepEx loop you showed is not guaranteed to collect all cancellations.

Keep track for each socket of the pending operations, and use WaitForSingleObjectEx instead of SleepEx, to wait explicitly for each one.

Ben Voigt
  • 277,958
  • 43
  • 419
  • 720
  • Ok. So for each outstanding WSARecv and WSASend I keep a bookkeeping and when the callbacks have returned can I cleanup the overlapped structures and the socket wrapper object. But this means there must be a guarantee that the callback will be called or else I might wait indefinitely for the callback and hence will have a memory leak instead. I could not find if this is guaranteed by the OS. Is this guaranteed? – Ritsaert Hornstra Jul 26 '18 at 16:39
  • If you got `WSAEWOULDBLOCK` from the initial call, you are guaranteed to get the callback when the operation ends, whether success, failure, or cancellation. – Ben Voigt Jul 26 '18 at 17:07
  • From MSDN: both an 0 (OK) and WSAEWOULDBLOCK returns a callback guaranteed. So bookkeeping it is. – Ritsaert Hornstra Jul 26 '18 at 18:10