4

When calling WSASend(), I have to pass it a WSAOVERLAPPED instance, and I cannot re-use this WSAOVERLAPPED instance until the previous WSASend() operation has been completed (i.e. when a completion packet has been placed in the completion port, and when I deque this completion packet I guess).

Based on this understanding, I have a WSAOVERLAPPED instance associated with each socket in my application, and I also have a boolean variable (called is_sending_in_progress).

Now let's say that I have a button that will send the string "hello" when clicked to the other side.

Now when the user clicks on this button, I will check to see if is_sending_in_progress is false, and if it is false, then I would call WSASend() and then set is_sending_in_progress to true, now when I call GetQueuedCompletionStatus() to deque the completion packet, I set is_sending_in_progress to false. And if the user clicks on the button and is_sending_in_progress was true, then I would display a message box telling the user that he can't send anything right now until the previous send operation is completed.

Now I don't think that this is a good approach to handle the sending of data in IOCP, because the user would get this message box a lot (especially if the IOCP threads are busy and it would take them some time to set is_sending_in_progress to false).

So is there a better approach to handling the sending of data in IOCP, like having multiple WSAOVERLAPPED instances for each socket, and using the WSAOVERLAPPED instance available when calling WSASend()?

Steve
  • 691
  • 5
  • 18

3 Answers3

2

you are completely wrong understand IOCP and asynchronous I/O.

I have a WSAOVERLAPPED instance associated with each socket in my application

NO!!!

you can have any class/structure associated/encapsulated socket handle. but for every I/O operation you must allocate some another data structure inherited from OVERLAPPED. but need clear understand - this is structure per operation but not per socket . this instance must be allocated just before I/O operation begin. and destroyed just after I/O operation is end.

this structure is somehow related to IRP and have similar sense and lifetime. in this structure except OVERLAPPED you must pointer to your class instance which is encapsulate socket, some tag which is described what is this I/O operation - send, receive, connect, disconnect, etc.. and possible some additional data - related to operation

and I also have a boolean variable (called is_sending_in_progress).

again NO!!!

we can have multiple I/O operation on same socket at time. we can have multiple send operation in time on same socket - of course for every operation must be unique OVERLAPPED (your custom user mode IRP) but need clear understand - this OVERLAPPED per operation only - class instance where socket handle located - must not containing (inherit from) OVERLAPPED. we can have send and receive operation in same time. receive and disconnect.

only single restriction - receive operation can not be multiple (several) at some time. but this not OVERLAPPED restriction - simply if you got 2 packets of data at once - you can not know which was send first and which second - so you lost data order

really asynchronous I/O give your very big freedom and power in action, but only if you deep understand it.

So is there a better approach to handling the sending of data in IOCP

when you using asynchronous I/O with IOCP we have callback called when operation is finished ( FileIOCompletionRoutine or IoCompletionCallback - this callbacks will be called automatically by system when operation is complete or if you yourself call GetQueuedCompletionStatus - you need yourself also call this callback. and all operations with socket we must do inside this callback. if we need send big portion of data - we can break it on chunks. send first chunk direct. and when send will be complete - callback is called - and here we call send next chunk of data.. - send next chunk exactly when previous send is complete.

RbMm
  • 31,280
  • 3
  • 35
  • 56
  • IOCP absolutely does not care how you allocate your `OVERLAPPED` structures. You can re-use one over and over again so long as you zero-initialize it. – Cory Nelson Feb 24 '17 at 19:23
  • *"this is structure per operation but not per socket"* Yes, I know. I associated the `WSAOVERLAPPED` instance with my socket so that when I want to call `WSASend()` on this socket, I know which `WSAOVERLAPPED` instance to use (the one associated with this socket!). – Steve Feb 24 '17 at 19:29
  • @CoryNelson - yes, but how this is related to my answer ? – RbMm Feb 24 '17 at 19:29
  • "this instance must be allocated just before I/O operation begin. and destroyed just after I/O operation is end." – Cory Nelson Feb 24 '17 at 19:29
  • @Steve - no, look like you not understand. `I associated the WSAOVERLAPPED instance with my socket ` - this is your **fundamental** error – RbMm Feb 24 '17 at 19:30
  • @CoryNelson - yes. "this instance must be allocated just before I/O operation begin. and destroyed just after I/O operation is end." - can only repeat this. but how we allocated this instance containing overlapped - our task – RbMm Feb 24 '17 at 19:31
  • @CoryNelson - look like you also noting understand in asynchronous I/O :) – RbMm Feb 24 '17 at 19:32
  • 1
    @RbMm I don't think you fully understood what I said in my question. I know that the `WSAOVERLAPPED` instance must be uniquely used with each operation, that's the point of the `is_sending_in_progress` variable, so that I would not re-use the `WSAOVERLAPPED` instance until I know that the previous send operation has ended! – Steve Feb 24 '17 at 19:33
  • @Steve - reread what I write. you must allocate overlapped data per operation but not `associated the WSAOVERLAPPED instance with my socket`. and for `know which WSAOVERLAPPED instance to use ` - you need have **pointer** to your socket class in instance inherited from overlapped – RbMm Feb 24 '17 at 19:34
  • @RbMm I am not using C++, I am using C. And a `WSAOVERLAPPED` can't be associated with a socket. This is not what I meant when I wrote this sentence! – Steve Feb 24 '17 at 19:36
  • @Steve - from your comments I view that you still completely nothing understand. again (last time) - OVERLAPPED must be allocated per every I/O operation. instance where socket handle located - must not be associated / containing overlapped - this is huge error. you can have multiple io operations at same time.. – RbMm Feb 24 '17 at 19:38
  • @RbMm I still think that you did not fully understand what I said in my question. – Steve Feb 24 '17 at 19:39
  • @Steve - you can have multiple send at same time and you not need call `GetQueuedCompletionStatus() ` at all. if you need send sequence big data - you can break it on chunks and send next chunk when previous send operation is finished - in callback - but not need have any variable like `is_sending_in_progress` – RbMm Feb 24 '17 at 19:46
1

You are over thinking this.
when initiating a new asynchronous IO, allocate a new OVERLAPPED structure, receive it back in the IOCP , then delete it.

a very popular approach is to "inherit" (as much as C provides - compose it with another variable) from OVERLAPPED and attach a callback to it. when IOCP dequeues the OVERLAPPED , dispatch the callback to some thread-pool and delete the OVERLAPPED.

You might also not want to use raw Win32 , but to find an abstraction on top of it, like boost.asio (C++). you might also want to use something more user friendly than raw sockets, like HTTP requests.

David Haim
  • 25,446
  • 3
  • 44
  • 78
  • Is there a limit on the number of pending `WSASend()` calls? – Steve Feb 24 '17 at 21:27
  • I remember reading somewhere that I should not have many pending `WSASend()` calls or something, not sure, I can't find where I read that. – Steve Feb 24 '17 at 22:13
  • 1
    @Steve the only limit is the system limit, AKA the available memory, the CPU and network card. there is no hard coded value, nor the system has any reason to limit you from doing many sending – David Haim Feb 24 '17 at 22:48
  • Each send uses up a small amount of system resources. So there's a limit to the number of sends that's based on the amount of resources used and the total amount of resources available. One of these resources is non-paged pool memory and prior to Windows Vista this was in pretty short supply. On modern Windows operating systems the amount of non-paged pool is directly related to the amount of memory in the system. See here http://www.serverframework.com/asynchronousevents/2011/06/tcp-flow-control-and-asynchronous-writes.html for why it's a good idea to keep track of how many sends are pending – Len Holgate Feb 26 '17 at 16:12
  • @Len Holgate Let's say I am using Windows XP on a PC with 2 GB of memory, what is the approximate number of pending `WSASend()` requests that my application can have? – Steve Feb 26 '17 at 22:05
  • It will depend on what else is using non-paged pool at the time. There is no answer to this question as the value would change depending on what else is running and using non-paged pool at the same time. You would need to run tests on the target box and then limit the number of concurrent connections and pending sends accordingly. Ideally you'd never get anywhere near the limit. Ideally you aren't supporting XP anymore. – Len Holgate Feb 27 '17 at 13:45
0

It sounds like your problem is not one of IOCP, but of general buffering strategy. A workflow that might help you would be something like:

// on button click

add_message_to_buffer();
if(!is_sending_in_progress)
{
    is_sending_in_progress = true;
    start_send();
}

// in your IOCP loop, on completion of send.

if(buffer_has_more_data)
{
    start_send();
}
else
{
    is_sending_in_progress = false;
}
Cory Nelson
  • 29,236
  • 5
  • 72
  • 110
  • when we using asynchronous I/O with IOCP we really have callback called when operation (send for example) is finished. so need send next chunk of data exactly in this callback, but not need have variable `is_sending_in_progress` – RbMm Feb 24 '17 at 19:48
  • @RbMm you aren't understanding the problem, and my answer already addresses that. – Cory Nelson Feb 24 '17 at 21:05
  • I think your approach is great, but when I want to send a 10 GB file, I should not use this approach, correct? because if I used this approach, then I would have to put a 10 GB of data in memory (which is a terrible idea, also it would not work since a 32-bit process is only allowed to have 2 GB of memory). So I think a better approach for sending a file is to send the first chunk of the file, and then wait for the completion packet, and then send the next chunk, etc. until all of the file is sent. I should probably use another socket for the sending of files. Is this approach correct? – Steve Feb 24 '17 at 21:53
  • @Steve - you need use the same socket for sending this file – RbMm Feb 24 '17 at 22:53
  • @Steve correct, it'd really only work if your messages are short -- i.e. a chat client or a game or something. If you're going to send a 10GB file, I would have 1 outstanding read i/o going on concurrently with the send i/o. so READ A -> SEND A / READ B -> SEND B / READ C – Cory Nelson Feb 25 '17 at 01:12
  • @CoryNelson- read and send absolute independed operation. socket can read and send at same time – RbMm Feb 25 '17 at 01:27