IOCP thread processing

Question

I am making an IOCP server in C#, invoking win apis. I have 1 accept thread connection and worker threads depending of CPU cores.

My problem is, i am getting 2 threads trying to process the same data at the same time, based on the following snippets can you tell me whats the solution or is this a design problem?

public class WorkerThread
    {
        public void Worker(NetSharedData data)
        {
            IntPtr bytestransfer = Marshal.AllocHGlobal(sizeof(UInt32));
            IntPtr clientid = Marshal.AllocHGlobal(sizeof(UInt32));
            IntPtr poverlapped = Marshal.AllocHGlobal(sizeof(UInt32));
            Console.WriteLine("Worker thread running");
            while (true)
            {
                Marshal.WriteInt32(clientid, 0);
                Marshal.WriteInt32(bytestransfer, 0);
                if (SocketInvoke.GetQueuedCompletionStatus(data.completionport, bytestransfer, clientid, poverlapped, SocketInvoke.INFINITE) == true)
                {
                        if (Marshal.ReadInt32(poverlapped) == 0)
                        {
                            //thread shutdown
                            Console.WriteLine("Worker thread shutdown");
                            break;
                        }
                        Client client = data.Clients[Marshal.ReadInt32(clientid)];
                        if (Marshal.ReadInt32(bytestransfer) != 0)
                        {
                            if (client.operationtype == SocketInvoke.FD_WRITE)
                            {
                                if (client.send_data.ispending_operation == true)
                                {
                                    client.SendLeft((uint)Marshal.ReadInt32(bytestransfer));
                                    break;
                                }
                            }
                            if (client.operationtype == SocketInvoke.FD_READ)
                            {
                                if (!client.Recv((uint)Marshal.ReadInt32(bytestransfer)))
                                {
                                    client.Close();
                                    break;
                                }
                                if (data.OnRecv(client) == true)
                                {
                                    //SendLeft test
                                }
                            }
                        }
                }
            }
        }
    }

/*
//THIS IS A CODE SNIPPET THAT BELONGS TO THE ACCEPT THREAD CLASS
*/

public virtual bool Recv(uint expected_data_transfer)
        {
            IntPtr pwsabuf;
            IntPtr wsaoverlapped;
            IntPtr bytes_recv = Marshal.AllocHGlobal(sizeof(int));
            IntPtr flags = Marshal.AllocHGlobal(sizeof(int));
            Marshal.WriteInt32(flags, 0);
            set_total_transfer(SocketInvoke.FD_READ, expected_data_transfer);
            pwsabuf = recv_data.rtn_wsabuffarray(recv_data.buffer, (uint)recv_data.bufflen);
            wsaoverlapped = Marshal.AllocHGlobal(Marshal.SizeOf(typeof(SocketInvoke.WSAOVERLAPPED)));
            Marshal.StructureToPtr(overlapped, wsaoverlapped, false);
            SocketInvoke.FillMemory(wsaoverlapped, (uint)Marshal.SizeOf(typeof(SocketInvoke.WSAOVERLAPPED)), 0);
            unsafe
            {
                setoptype(SocketInvoke.FD_READ);
                if (SocketInvoke.WSARecv(Convert.ToUInt32(sock.Handle.ToInt32()), pwsabuf,
                    (uint)1, bytes_recv, flags, wsaoverlapped, (IntPtr)null) == SocketInvoke.SOCKET_ERROR)
                {
                    if (Marshal.GetLastWin32Error() != SocketInvoke.WSA_IO_PENDING)
                    {
                        return false;
                    }
                    return true;
                }
            }
            return true;
        }

Also is it possible to get a partial send or partial recv of data using WSASend and WSARecv?.

1) WHY are you doing this? Either use the managed async socket methods or do the whole of the socket code in C or C++ and expose it to managed code via a nice clean interface. 2) You need a unique overlapped structure per read or write operation. — Len Holgate, Nov 24 '12 at 10:34

score 4 · Answer 1 · answered Nov 24 '12 at 10:39

1) WHY are you doing this? Either use the managed async socket methods or do the whole of the socket code in C or C++ and expose it to managed code via a nice clean interface.

2) I'm guessing here because the code is unclear, but... You need a unique overlapped structure per read or write operation.

3) Yes WSASend can fail and have sent part of the data, usually only if your data buffer is larger than the OS page size and you hit the locked pages limit - OR if you run out of non-paged pool. If this happens then there is possibly nothing you can do to recover from it - try not to get into that situation in the first place by placing limits on the number of connections you can handle. Expect to be able to handle 100,000s connections but watch how many overlapped operations you have pending.

4) WSARecv, like all TCP read operations, can return anywhere between 1 byte and the size of the buffer supplied. If it returns 0 then the peer has shut down its send side of the connection.

5) Use AcceptEx and get rid of that Accept thread. Having a dedicated Accept thread causes another thread to context switch to and fails to scale if you support listening on multiple ports from within one program.

IOCP thread processing

1 Answers1