Coding pattern for dependant overlapped io in windows

Question

I'm a linux programmer and recently involved in porting an epoll based client with two file descriptor written in c to windows.
As you know,in linux using epoll or select (I know windows supports select, but its not efficient at all) you can block on file descriptors until a file descriptor is ready and you can know when it is ready to write and when read.

I have taken a look at windows IOCP and it sounds ok for overlapped io in microsoft world. But in all samples it is used for a multi client server that each client's socket is independent from other sockets.

using completions ports, it can be done creating a completionKey structure for each client and put a variable in struct and make it read when invoking WSArecv and wirt when WSAsend and the other variable indicating socket value and retrieving them from GetQueuedCompletionStatus to know what to do, if write is done for socket, do read, and vise versa.

But in my case, the file descriptors (fd) are really overlapped. reading from one fd, makes read and write to other fd and that makes it hard to know what operation realy happend for each fd in GetQueuedCompletionStatus result because there is one completionKey associated for each fd. to be clear consider this please:

There is two handles called fd1 and fd2 and completionKey1 is holding handle and status for f1 and completionKey2 for fd2 and completionKey variable is for retrieving completion from GetQueuedCompletionStatus.

    GetQueuedCompletionStatus(port_handle, &completionKey.bufflen, (PULONG_PTR)&completionKey,(LPOVERLAPPED *)&ovl,INFINITE);

   switch (completionKey.status)
    {
        case READ:
            if(completionKey->handle == fd1)
            {
                fd1_read_is_done(completionKey.buffer,completionKey.bufflen);
                completionKey->status = WRITE;
                do_fd1_write(completionKey);
                completionKey2->status = WRITE;
                completionKey2->buffer = "somedata";
                do_fd2_write(completionKey2);
            }
            else if(completionKey->handle == fd2)
            {
                fd2_read_is_done(completionKey.buffer,completionKey.bufflen);
                completionKey->status = WRITE;
                do_fd2_write(completionKey);
                completionKey1->status = WRITE;
                completionKey1->buffer = "somedata";
                do_fd1_write(completionKey1);
            }
            break;
        case WRITE_EVENT:
            if(completionKey->handle == fd1)
            {
                fd1_write_is_done(completionKey.bufflen);
                completionKey->status = READ;
                do_fd1_read(completionKey);
                completionKey2->status = READ;
                do_fd2_read(completionKey2);
            }
            else if(completionKey->handle == fd2)
            {
                fd2_write_is_done(completionKey.bufflen);
                completionKey->status = READ;
                do_fd2_read(completionKey);
                completionKey1->status = READ;
                do_fd1_read(completionKey1);
            }
            break;
    }

in the above code, it comes a situation that some of altering completionKeys will override the pending reads or writes and the resulted completionKey->status would be wrong (it will report read instead of write for instance) and worst is the buffer will override. if I use locking for completionKeys, it will lead to dead lock situations.

After looking to WSAsend or WSArecv, noticed there is a overlap parameter can be set for every send or receive. but it leads to two major problems. according to WSAOVERLAPPED structure:

    typedef struct _WSAOVERLAPPED {
  ULONG_PTR Internal;
  ULONG_PTR InternalHigh;
  union {
    struct {
      DWORD Offset;
      DWORD OffsetHigh;
    };
    PVOID  Pointer;
  };
  HANDLE    hEvent;
} WSAOVERLAPPED, *LPWSAOVERLAPPED;

First, there is no place for putting status and appropriate buffer in it and most of them are reserved.

Second if could make a work for first problem, I need to check if there is no available overlapped left and all of them are used in pending operations, allocate a new one for every read and write and because of the client is going to be so busy, it might happens a lot and besides, managing those overlapped pools is a headache. so am I missing something or microsoft has screwed this?

And because of I don't need multithreading, is there another way to solve my problem?
thanks in advance
Edit
As I guessed, the first problem that I mentioned in using overlapped struct has answer and I need just create another struct with all buffers and status and etc and put OVERLAPPED as first filed. now you solve me the others ;)

Answer 1

You're really asking two different questions here. I can't answer the first one, as I've never used IO completion ports, but from everything I've read they're best avoided by everyone but experts. (I will point out an obvious solution to the problem I think you're describing: rather than actually writing the data to the other socket while another write is still pending, put the data in a queue and write it later. You still have to deal with two simultaneous operations on a given socket - one read and one write - but that shouldn't be a problem.)

However, it's easy to use OVERLAPPED (or WSAOVERLAPPED) structures to track the status of overlapped requests. All you do is embed the OVERLAPPED structure as the first element in a larger structure:

typedef struct _MyOverlapped
{
  WSAOVERLAPPED overlapped;
  ... your data goes here ...
} MyOverlapped, lpMyOverlapped;

then cast the LPWSAOVERLAPPED sent to the completion routine to lpMyOverlapped to access your context data.

Alternatively, if you are using a completion routine, the hEvent member of WSAOVERLAPPED is guaranteed to be unused, so you can set this to a pointer to a structure of your choice.

I don't see why you think that managing the pool of overlapped structures is going to be a problem. There's exactly one overlapped structure per active buffer, so every time you allocate a buffer, allocate a corresponding overlapped structure.