intensive data sending breaks IOCP events

Question

I use Windows IOCP in my app for few years without any issues (permanent TCP communication with small messages), but recently I had to send very big application state (25k of messages, about 12Mb all together).

I use 0x2000-bytes buffers, combined with IOCP events: first write triggers WSASend, all other just fill the buffers. When GetQueuedCompletionStatus gets event, that write was completed, next buffer is sent via WSASend, and so on.

This is first time I use such a use-case to send so big data at once. And I noticed, that after some time GetQueuedCompletionStatus stops receiving events.

Here is visualization, how data from buffer is sent from IOCP callback:

14:16:19.894, WSASend scheduled buffer size (3005) , pendingWrite bytes 3005
14:16:20.021, WSASend scheduled buffer size (1175) , pendingWrite bytes 1175
14:16:20.276, WSASend scheduled buffer size (4652) , pendingWrite bytes 4652
14:16:20.411, WSASend scheduled buffer size (2135) , pendingWrite bytes 2135
14:16:20.604, WSASend scheduled buffer size (3366) , pendingWrite bytes 3366
14:16:20.835, WSASend scheduled buffer size (3718) , pendingWrite bytes 3718
14:16:20.943, WSASend scheduled buffer size (665)  , pendingWrite bytes 665
14:16:21.087, WSASend scheduled buffer size (2931) , pendingWrite bytes 2931
14:16:21.109, WSASend scheduled buffer size (296)  , pendingWrite bytes 296
14:16:21.109, WSASend scheduled buffer size (21)   , pendingWrite bytes 21
14:16:21.182, WSASend scheduled buffer size (380)  , pendingWrite bytes 380
14:16:21.184, WSASend scheduled buffer size (160)  , pendingWrite bytes 160
14:16:21.355, WSASend scheduled buffer size (5885) , pendingWrite bytes 5885
14:16:21.531, WSASend scheduled buffer size (8169) , pendingWrite bytes 16135 <-- here buffers start growing
14:16:21.691, WSASend scheduled buffer size (8173) , pendingWrite bytes 19727
14:16:21.692, WSASend scheduled buffer size (8176) , pendingWrite bytes 11554

As you can see, after some moment, pendingWrite starts growing, but it's OK, I have a lot of memory for buffering it. But after last send (at 14:16:21.692) GetQueuedCompletionStatus simply does't trigger.

I tried to increase send/receive socket buffers (SO_RCVBUF/SO_SNDBUF) to 80Kb, but looks like this didn't have any effect.

I have no idea, what should I check next, WSASend doesn't return any error (only ERROR_IO_PENDING which is expected).

PS: communication is on the same PC, Windows 7 x64.

without see the code think not possible say something – RbMm Aug 25 '19 at 08:54 — RbMm, Aug 25 '19 at 08:54

score 0 · Accepted Answer · answered Sep 06 '19 at 07:18

Shortly: my assumption was, that code worked well, so I guessed, that IOCP (or socket) settings were unsuitable for such a throughput, or maybe there are some limitations for socket buffer.

Answer: code had race condition, so sometime receiver stopped consuming data from socket, that's why sending socket's buffer finally reached it's limit, and GetQueuedCompletionStatus didn't trigger new events for socket's write callback.

Summary: IOCP works as expected, no extra configuration is needed.

When race was fixed, I was able to send over 150k small packets (20-200 bytes) in a moment.

intensive data sending breaks IOCP events

1 Answers1