Sending syncronously with System.Net.WebSockets.SendAsync

Question

I'm all in favour of multithreading and asyncronous tasks, but letting go of control isn't my cup of tea.

I'm working on a project where one process is communicating with multiple targets over websockets, and sending should obviously be as speedy as possible, but has to be synced with each other. Think in line of streaming a videostream to multiple devices simontaniously. Thus, I don't want to continue sending the next frame before the first one has been receieved by the target.

When using .net WebSockets, I've tried this both with HttpListner and the Kestrel implementation (which I guess is more or less the same), then the socket.SendAsync seems to be receiving the data and buffering it locally before sending it, returning TaskCompleted before its actually sent. This means that the send task basically completes instantly.

I "could" of course implement an ACK schema where the recipient sends an ACK when the whole packet has been received, but that would in itself slow down the process, which I would like to avoid as this is already known by the system itself, as it does try to send the queued up packages continously.

This is basically the code responsible for sending the data, in chunks of 2048b:

Log.Info($"Send {sendBytes.Length}b");
if (socket!.State == WebSocketState.Open)
{
    await socket.SendAsync(
        sendBuffer,
        WebSocketMessageType.Binary,
        endOfMessage: true,
        cancellationToken: cts
        );
}
Log.Info($"Done "); 

I would assume the await here would actually wait for the transmission to complete, which is doesn't do, but it seems to await the task completedness.

And on the recipient side this is processing the data, with the debug responses before and after:

wsClientSession.send("REC " + (String)payloadLength + "b");
ProcessFrame(payload, payloadLength);
wsClientSession.send("ACK " + (String)payloadLength + "b");

The logging on the sender side shows this:

• Thread 28 is the sending thread, which completes impossibly fast. It needs to send 6198 bytes and chuncks this up in 4 transmissions. It is obviously all done before the first packet has left the server.
• The remaining threads are the async recipients of the debug responses.
• (size differences are due to the encryption of the data)

2022-02-07 16:03:34.8887 threadid:28 # Info # Send 6198b to 5d8263c6-909c-47a1-b4d0-57d74d6a4665
2022-02-07 16:03:34.8887 threadid:28 # Info # Send 2048b
2022-02-07 16:03:34.8887 threadid:28 # Info # Done
2022-02-07 16:03:34.8887 threadid:28 # Info # Send 2048b
2022-02-07 16:03:34.8887 threadid:28 # Info # Done
2022-02-07 16:03:34.8887 threadid:28 # Info # Send 2048b
2022-02-07 16:03:34.8887 threadid:28 # Info # Done
2022-02-07 16:03:34.8887 threadid:28 # Info # Send 128b
2022-02-07 16:03:34.8887 threadid:28 # Info # Done
2022-02-07 16:03:34.8887 threadid:28 # Info # Send done to 5d8263c6-909c-47a1-b4d0-57d74d6a4665
2022-02-07 16:03:35.0024 threadid:7 # Info # 5d8263c6-909c-47a1-b4d0-57d74d6a4665 - REC 2030b
2022-02-07 16:03:35.0090 threadid:7 # Info # 5d8263c6-909c-47a1-b4d0-57d74d6a4665 - ACK 2030b
2022-02-07 16:03:35.0090 threadid:17 # Info # 5d8263c6-909c-47a1-b4d0-57d74d6a4665 - REC 2030b
2022-02-07 16:03:35.0260 threadid:33 # Info # 5d8263c6-909c-47a1-b4d0-57d74d6a4665 - ACK 2030b
2022-02-07 16:03:35.0260 threadid:33 # Info # 5d8263c6-909c-47a1-b4d0-57d74d6a4665 - REC 2030b
2022-02-07 16:03:35.0260 threadid:30 # Info # 5d8263c6-909c-47a1-b4d0-57d74d6a4665 - ACK 2030b
2022-02-07 16:03:35.0373 threadid:17 # Info # 5d8263c6-909c-47a1-b4d0-57d74d6a4665 - REC 108b
2022-02-07 16:03:35.0373 threadid:4 # Info # 5d8263c6-909c-47a1-b4d0-57d74d6a4665 - ACK 108b

Now, is my understanding of this WebSocket implementation all wrong? Is it not possible to have a callback/event or similar on data sent completedness? Can I achieve this in another way? Executing this true synchronously would be a solution, but I don't seem to be able to force that.

GAAH!!

Answer 1

I'm working on a project where one process is communicating with multiple targets over websockets, and sending should obviously be as speedy as possible, but has to be synced with each other. Think in line of streaming a videostream to multiple devices simontaniously. Thus, I don't want to continue sending the next frame before the first one has been receieved by the target.

Well... actually... both video and audio transmissions tend to use UDP these days and are deliberately designed to allow dropped packets and frames. Missing a single frame in a 30 (or 60) fps source is barely noticeable, but putting a spinner over the video while re-fetching an old frame is very noticeable.

So, perhaps the first thing to consider is that you might want (or need) to relax your requirements. Networking is not the same as hardwired controls, and any solution like this is distributed by nature and therefore must have either some eventual consistency or some partition intolerance (see: CAP theorem).

the socket.SendAsync seems to be receiving the data and buffering it locally before sending it, returning TaskCompleted before its actually sent. This means that the send task basically completes instantly.

This is not just websockets, or .NET, or even Windows. This is (deliberately) how all TCP/IP APIs work. The "send" is considered complete when it reaches the OS buffers. In the TCP/IP world, you are guaranteed that the data will eventually reach the recipient (possibly after retries), or that your socket will eventually enter an error state.

I "could" of course implement an ACK schema where the recipient sends an ACK when the whole packet has been received, but that would in itself slow down the process, which I would like to avoid as this is already known by the system itself, as it does try to send the queued up packages continously.

Websockets do have a message abstraction and maintains message boundaries for you. But websockets are implemented over TCP/IP, which is a stream abstraction (not message or packet). The network itself uses a packet abstraction, and packets are where the ACK/RST/etc come into play. So, the OS is aware of packet-level ACKs/etc and will remove bytes from the TCP/IP stream buffer. But websockets do not have a message-level ACK, so there's no way to know when a message has been received.

Websockets are built on TCP/IP, so you get the same guarantee: you know that a sent message will either (eventually) be received, or (eventually) you will be notified of an error. That's pretty much all you get; the only major difference with TCP/IP is that you get a message abstraction instead of a stream abstraction.

So, your options are:

• Accept the eventual consistency. Send messages to all devices simultaneously, and accept that some updates may be delayed.
• Accept the partition intolerance. Add an ACK response message, and only send the next update after all devices have ACKed. (This doesn't tolerate partitioning because if one device is disconnected, they all break).

If you do go with the ACK-lockstep approach, you might want to consider using UDP/IP instead of a TCP/IP-based protocol, but that is a whole ton of work.

Executing this true synchronously would be a solution, but I don't seem to be able to force that.

Synchronous APIs are not a solution. There's no API that does what you want; all TCP/IP writes everywhere consider the write "complete" when it reaches the OS; this is true regardless of sync/async, language, runtime, or OS.