0

I create udp socket a wait for incoming packets, when receive any packet - create new socket, bind to the same address and connect to the client address(from recvfrom).

I can send data to this socket(client gets packets), but when client sends packets, this 'connected' socket don't get any packets, but first socket does.

The same code works well on Unix.

auto masterSocket = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
setsockopt(masterSocket , SOL_SOCKET, SO_REUSEADDR, /* 1 */);
bind(masterSocket, {INADDR_ANY, 30000});

while (true) {
    recvfrom(masterSocket, /* address */);

    auto clientSocket = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
    setsockopt(clientSocket , SOL_SOCKET, SO_REUSEADDR, /* 1 */);
    bind(clientSocket, {INADDR_ANY, 30000});
    connect(clientSocket, /* address from recvfrom */);

    send(clientSocket, /* some data */); // this works well, data is delivered to the client
    int res = recv(clientSocket, /* buff */); // this stucks on windows, but returns on linux
    int res = recvfrom(masterSocket, /* buff */); // this works on both windows and linux
}

This is pseudocode, I can send both client and server source if need. But problem is that program works as expected on linux, but on windows 'connected' udp socket didn't get incoming packets...

Lawliet
  • 1
  • 1
  • 4
  • From https://stackoverflow.com/a/14388707/1269466 link: https://learn.microsoft.com/de-de/windows/win32/winsock/using-so-reuseaddr-and-so-exclusiveaddruse#using-so_reuseaddr I think this kind of sharing of the same port is nondeterminant. The OS is free to choose either of the sockets for a packet and ignore the other. – lossleader Sep 05 '19 at 23:06
  • This way DTLS works, no? Creating master socket and client socket with the same port – Lawliet Sep 05 '19 at 23:47
  • I wouldn't use `INADDR_ANY` for the connected socket. Using `WSARecvMsg()` instead of `recvfrom()`, you can determine the local IP that received the client's packet, and then you can `bind()` the connected socket to that same IP. But really, why do you need to use `connect()` at all? Why not just keep a running list of known clients, and then route packets from `recvfrom()` through your code as needed based on which client `recvfrom()` reports? – Remy Lebeau Sep 06 '19 at 00:14
  • Because in this case we will have one socket per server or socket + reading/writing thread per core + global lock for hashtable with addr -> session mapping to get maximum throughput. As I understand, one socket can be bottleneck in highload server, so we need to create socket per thread, but if we do that, we will have continuation on hashtable. Best option will be to do socket per 'connection', so when we receive packet on socket we already know which session it is, we don't need to lookup to hashtable with sessions – Lawliet Sep 06 '19 at 09:28
  • your server and client are both bound to 30000 so they are basically a conflict waiting to happen with more than 1 instance of any part on the same host. If you look at most protocols they either allow the initaitor to get a free port, or they have 2 separate reserved ports. https://github.com/nplab/DTLS-Examples/blob/226f222e528858b3a8c5fa3326b0599d25d3ef1c/src/dtls_udp_echo.c#L764 – lossleader Sep 06 '19 at 18:18
  • 1
    my client use 0 port, so system will give some free port. – Lawliet Sep 06 '19 at 20:02
  • In solution you sent - https://github.com/nplab/DTLS-Examples/blob/226f222e528858b3a8c5fa3326b0599d25d3ef1c/src/dtls_udp_echo.c#L658 https://github.com/nplab/DTLS-Examples/blob/226f222e528858b3a8c5fa3326b0599d25d3ef1c/src/dtls_udp_echo.c#L405 first - creating master socket that binds to some addr, second - creating socket per client that binds to the same addr and connects to the client addr. btw, this solution doesn't work on Windows, but works on linux – Lawliet Sep 06 '19 at 20:05
  • I misinterpreted your code as trying to test problems locally between the 2 sockets instead of trying to shed connections from a listening socket. I think you are correct that this doesn't work on windows.. it seems like it could potentially work if you retreived a local address from recvmsg to make the 2nd socket bind more specific, but the msdn article only hints at some understanding of specificity during bind from its tables, while it sounds like they claim to have a totally indeterminant fanout in unicast.. – lossleader Sep 06 '19 at 21:50
  • So, no chance to 'fix' this problem on windows? How can I retrieve more specific local address from recvmsg to bind? As I understand docs, I can only get remote address and use it in connect. Thank you – Lawliet Sep 06 '19 at 22:49
  • @Lawliet "*As I understand, one socket can be bottleneck in highload server, so we need to create socket per thread*" - plenty of high-performance servers use a single receiving socket. For high performance, consider using overlapped I/O, completion port, or registered I/O for the reading. You can still have separate threads per cpu that pull data off a receiving socket quickly. "*so when we receive packet on socket we already know which session it is*" - a session that is based on the remote ip/port reported by `recvfrom()` can accomplish that same thing, no hashtable needed. – Remy Lebeau Sep 06 '19 at 23:49
  • @Lawliet "*How can I retrieve more specific local address from recvmsg to bind?*" - enable the `IP_PKTINFO` option on the socket, then `WSARecvMsg()` will report a `in_pktinfo` struct in the `WSAMSG` buffer it outputs, containing info about the dest IP the packet was sent to, and the network interface the packet was received on. – Remy Lebeau Sep 06 '19 at 23:51
  • @RemyLebeau "a session that is based on the remote ip/port reported by recvfrom() can accomplish that same thing, no hashtable needed", well, I mean the real session object in c. So, if we are using socket per client then we can store this info inside socket(I don't remember how, but we can provide user data as pointer). And later, when we receive something into this socket we get session object simple from socket handler, no need to lookup hashtable with addr -> session. – Lawliet Sep 07 '19 at 00:09
  • Tried in_pktinfo, it returns 127.0.0.1 (I connected from local machine), I can bind to this ip, and socket receive all data, but master socket will be blocked and it can't receive any new data... – Lawliet Sep 07 '19 at 00:52
  • @Lawliet there is no option to store user data in a socket. Some higher level framework wrappers allow that, like socket.io, but certainly not on the socket itself. – Remy Lebeau Sep 07 '19 at 02:16
  • @RemyLebeau I mean, not socket itself, but something like epoll can return struct to where we can store our pointer and socket handler and we can wait for events on this struct and get back session pointer immediately. This way we can create N client sockets for each new 'connection', asociate session object with each and on every recv even we alredy know which session it is, we don't need to lookup to table addr -> session – Lawliet Sep 07 '19 at 10:23
  • That why I won't to do it via one socket and recvfrom/sendto. Not only because of global queue in kernel space, but because I need to search session by addr in app logic. I can create thread per cpu with socket per thread, this will work, but then I need to have mutex to this hashtable. But if I use socket handler per client with bind/connect - I can store session pointer inside epoll_data struct(for example), and get it back when there is data on socket handler. So, no continuation even in multithread system, because one socket handler will receive only packets from one client – Lawliet Sep 07 '19 at 10:38
  • The trouble is that this requires a more complex fanout in the kernel, an windows claims they don't try to provide that.. so even if you get it working in a particular version, I wouldn't count on it contiuing. This question actually has references to the part of linux that tries to use remoteaddr info in a 4 tupple for more complicated fanout, and claims occassional problems happen on linux too: https://stackoverflow.com/questions/56683832/emulating-tcp-using-bind-connect-on-udp-sockets-with-so-reuseport – lossleader Sep 07 '19 at 11:39
  • So, better to not use this even on linux? What should I use then? One socket on server with recvfrom/sendto and hashtable with adrr -> session, right? Will I get something if I use thread per core with socket per thread + mutex on this hashtable? – Lawliet Sep 07 '19 at 11:50
  • You would essentially be building your own distributed re-fanout in userland, which would probably help if your task is mostly io bound and has a fair amount of data per transaction and/or longevity.. I would build the synchronization to split most things into distinct worker and network threads first and then determine which type needs to support a pool of >1 threads first. – lossleader Sep 07 '19 at 13:13
  • @Lawliet on Windows, Overlapped IO and IO Completion Ports allow you to associate user data with individual socket read/write operations – Remy Lebeau Sep 07 '19 at 15:52
  • @RemyLebeau, okay, so, how I can manage multiple sessions with one server socket? Like, on the first packet from user{ip, port} we create session and increment packetCount on each next incoming packet + send packet back to user. If you can show basic code, I would be very grateful! – Lawliet Sep 08 '19 at 23:55

0 Answers0