Why am I losing bytes from my Socket InputStream?

Question

I'm creating a socket based system for communicating between desktop and mobile devices. I'm using a simple protocol for reading and writing data to the streams between devices which ultimately end up as bytes:

1. First 4 bytes represent an integer, which defines a type of command to be executed
2. Next 4 bytes represent an integer, which is the length of the rest of the bytes in the stream
3. The remaining bytes represent the payload data, the total number of which corresponds to the result of (2.)

I'm able to successfully strip off the first 4 bytes and resolve the command ok, then strip off the next 4 bytes and resolve the length correctly. The problem comes when I strip off the remaining bytes, some of them are missing, and they're missing from the front of the remaining data.

For example; if the command is 1, and the length is 50, then there should be 50 bytes left in the stream but there's only 46 and it's bytes 0-3 which are missing.

The starting data is as follows:

• command: 1
• length: 50
• payload: C:\Users\dave\Music\Offaiah-Trouble_(Club_Mix).mp3

After converting this to a byte array, I get:

"\u0001\0\0\02\0\0\0C:\Users\dave\Music\Offaiah-Trouble_(Club_Mix).mp3"

(Question - why does the first integer get \u000 in front of it and the second does not?)

Here's some snippets of the code I'm using to parse this:

IBuffer inbuffer = new Windows.Storage.Streams.Buffer(4);
await _socket.InputStream.ReadAsync(inbuffer, 4, InputStreamOptions.None);
int command = BitConverter.ToInt32(inbuffer.ToArray(), 0);

The inbuffer at this point contains: "\u0001\0\0\0", and the BitConverter resolves this to 1

inbuffer = new Windows.Storage.Streams.Buffer(4);
await _socket.InputStream.ReadAsync(inbuffer, 4, InputStreamOptions.None);
int length = BitConverter.ToInt32(inbuffer.ToArray(), 0);

The inbuffer now contains: "2\0\0\0", and the BitConverter resolves this to "50"

inbuffer = new Windows.Storage.Streams.Buffer((uint)length);
await _socket.InputStream.ReadAsync(inbuffer, (uint)length, InputStreamOptions.Partial);
string path = Encoding.UTF8.GetString(inbuffer.ToArray());

The inbuffer now contains: "sers\dave\Music\Offaiah-Trouble_(Club_Mix).mp3"

Where did the missing "C:\U" go from the front of this?

Answer 1

So, I realised that I kept getting down-voted because the question wasn't concise and reproducible. So I created a small project to demonstrate just this part of the problem, and ironically enough the problem went away.

Here's the code:

public sealed partial class MainPage : Page
{
    private StreamSocketListener _listener;
    private StreamSocket _client;
    private StreamSocket _socket;
    private HostName _host;
    private int _port = 54321;

    public MainPage()
    {
        InitializeComponent();
        _listener = new StreamSocketListener();
        _listener.ConnectionReceived += async (sender, args) =>
        {
            _socket = args.Socket;
            await Receive();
        };
        _host = NetworkInformation.GetHostNames().FirstOrDefault(x => x.IPInformation != null && x.Type == HostNameType.Ipv4);
    }

    protected async override void OnNavigatedTo(NavigationEventArgs e)
    {
        base.OnNavigatedTo(e);
        await _listener.BindEndpointAsync(_host, $"{_port}");
        await Task.Run(async () =>
        {
            _client = new StreamSocket();
            await _client.ConnectAsync(_host, $"{_port}");

            int command = 1;
            byte[] cmd = BitConverter.GetBytes(command);
            byte[] path = Encoding.UTF8.GetBytes(@"C:\Users\Dave\Music\Offaiah-Trouble_(Club_Mix).mp3");
            byte[] length = BitConverter.GetBytes(path.Length);
            byte[] result = cmd.Concat(length.Concat(path)).ToArray();
            await _client.OutputStream.WriteAsync(result.AsBuffer());
        });
    }

    private async Task Receive()
    {
        while (true)
        {
            IBuffer inbuffer = new Windows.Storage.Streams.Buffer(4);
            await _socket.InputStream.ReadAsync(inbuffer, 4, InputStreamOptions.None);
            int command = BitConverter.ToInt32(inbuffer.ToArray(), 0);
            //The inbuffer at this point contains: "\u0001\0\0\0", and the BitConverter resolves this to 1

            inbuffer = new Windows.Storage.Streams.Buffer(4);
            await _socket.InputStream.ReadAsync(inbuffer, 4, InputStreamOptions.None);
            int length = BitConverter.ToInt32(inbuffer.ToArray(), 0);
            //The inbuffer now contains: "2\0\0\0", and the BitConverter resolves this to "50"

            inbuffer = new Windows.Storage.Streams.Buffer((uint)length);
            await _socket.InputStream.ReadAsync(inbuffer, (uint)length, InputStreamOptions.Partial);
            string path = Encoding.UTF8.GetString(inbuffer.ToArray());
        }
    }
}

If you create a new blank Universal project and run this, you'll see it produces the correct output.

Eventually I worked out that I had race conditions between this stream reader and another one. In the main project, I don't read all bytes in one go. I have a separate method reading and parsing the "command" before passing control off to another method to redirect control to one of several worker methods to service that particular command - stripping off the length and then the rest of the payload thereafter.

The problem was that the 'command reader' was then reading another 4 bytes from the stream, before the worker could read off it's payload.

Obviously the answer is that it should pause here and wait for the worker to finish, but I'm using async and await throughout so it surprised me that I was having this problem. The cause was a missing await, and the dreaded async void, as follows..

The offending code:

private async Task Listen()
{
    while (true)
    {
        //expects a 4 byte packet representing a command
        Debug.WriteLine($"Listening for socket command...");
        IBuffer inbuffer = new Windows.Storage.Streams.Buffer(4);
        await _socket.InputStream.ReadAsync(inbuffer, 4, InputStreamOptions.None);
        int command = BitConverter.ToInt32(inbuffer.ToArray(), 0);

        Debug.WriteLine($"Command received: {command}");
        ParseCommand(command);
    }
}

private async void ParseCommand(int command)
{
    //...
}

...and the ammended version:

private async Task Listen()
{
    while (true)
    {
        //expects a 4 byte packet representing a command
        Debug.WriteLine($"Listening for socket command...");
        IBuffer inbuffer = new Windows.Storage.Streams.Buffer(4);
        await _socket.InputStream.ReadAsync(inbuffer, 4, InputStreamOptions.None);
        int command = BitConverter.ToInt32(inbuffer.ToArray(), 0);

        Debug.WriteLine($"Command received: {command}");
        await ParseCommand(command);
    }
}

private async Task ParseCommand(int command)
{
    //...
}