Read continous bytestream from Stream using TcpClient and Reactive Extensions

Question

Consider the following code:

internal class Program
{
    private static void Main(string[] args)
    {
        var client = new TcpClient();
        client.ConnectAsync("localhost", 7105).Wait();
        var stream = client.GetStream();
        var observable = stream.ReadDataObservable().Repeat();

        var s = from d in observable.Buffer(4)
                let headerLength = IPAddress.NetworkToHostOrder(BitConverter.ToInt16(d.ToArray(), 2))
                let b = observable.Take(headerLength)
                select b.ToEnumerable().ToArray();
        s.Subscribe(a => Console.WriteLine("{0}", a));
        Console.ReadLine();
    }
}

public static class Extensions
{
    public static IObservable<byte> ReadDataObservable(this Stream stream)
    {
        return Observable.Defer(async () =>
        {
            var buffer = new byte[1024];
            var readBytes = await stream.ReadAsync(buffer, 0, buffer.Length);
            return buffer.Take(readBytes).ToObservable();
        });
    }
}

Basically I want to parse the messages I receive with Reactive Extensions. The header of the message is parsed correctly using the Buffer(4) and I get the length of the remainder of the message. The problem that arises is that when I do stream.Take(headerLength), the code reevaluates the whole "chain" and tries to get a new message from the stream instead of returning the rest of the bytes which already has been read from the stream. To be more exact, the first ReadAsync(...) returns 38 bytes, the Buffer(4) returns the first 4 of those, the observable.Take(headerLength) does not return the remainding 34 bytes but instead tries to read a new message with ReadAsync.

The question is, how can I make sure the observable.Take(headerLength) receives the already read 34 bytes and not try to read a new message from the stream? I've searched around for a solution, but I can't really figure out how to achieve this.

Edit: This solution (Using Reactive Extensions (Rx) for socket programming practical?) is not what I'm looking for. This isn't reading everything available in the stream (up to buffersize) and makes a continous bytestream out of it. To me this solution doesn't seem like a very efficient way to read from a stream, hence my question.

James World · Answer 1 · 2014-02-06T16:56:49.350

This approach isn't going to work. The problem is the way you are using the observable. Buffer will not read 4 bytes and quit, it will continually read 4 byte chunks. The Take forms a second subscription that will read overlapping bytes. You'll find it much easier to parse the stream directly into messages.

The following code makes a good deal of effort to clean up properly as well.

Assuming your Message is just this, (ToString added for testing):

public class Message
{
    public byte[] PayLoad;

    public override string ToString()
    {
        return Encoding.UTF8.GetString(PayLoad);
    }
}

And you have acquired a Stream then you can parse it as follows. First, a method to read an exact number of bytes from a stream:

public async static Task ReadExactBytesAsync(
    Stream stream, byte[] buffer, CancellationToken ct)
{
    var count = buffer.Length;
    var totalBytesRemaining = count;
    var totalBytesRead = 0;
    while (totalBytesRemaining != 0)
    {
        var bytesRead = await stream.ReadAsync(
            buffer, totalBytesRead, totalBytesRemaining, ct);
        ct.ThrowIfCancellationRequested();
        totalBytesRead += bytesRead;
        totalBytesRemaining -= bytesRead;
    }
}

Then the conversion of a stream to IObservable<Message>:

public static IObservable<Message> ReadMessages(
    Stream sourceStream,
    IScheduler scheduler = null)
{
    int subscribed = 0;
    scheduler = scheduler ?? Scheduler.Default;

    return Observable.Create<Message>(o =>
    {
        // first check there is only one subscriber
        // (multiple stream readers would cause havoc)
        int previous = Interlocked.CompareExchange(ref subscribed, 1, 0);

        if (previous != 0)
            o.OnError(new Exception(
                "Only one subscriber is allowed for each stream."));

        // we will return a disposable that cleans
        // up both the scheduled task below and
        // the source stream
        var dispose = new CompositeDisposable
        {
            Disposable.Create(sourceStream.Dispose)
        };

        // use async scheduling to get nice imperative code
        var schedule = scheduler.ScheduleAsync(async (ctrl, ct) =>
        {
            // store the header here each time
            var header = new byte[4];

            // loop until cancellation requested
            while (!ct.IsCancellationRequested)
            {                        
                try
                {
                    // read the exact number of bytes for a header
                    await ReadExactBytesAsync(sourceStream, header, ct);
                }
                catch (OperationCanceledException)
                {
                    throw;
                }
                catch (Exception ex)
                {
                    // pass through any problem in the stream and quit
                    o.OnError(new InvalidDataException("Error in stream.", ex));
                    return;
                }                   
                ct.ThrowIfCancellationRequested();

                var bodyLength = IPAddress.NetworkToHostOrder(
                    BitConverter.ToInt16(header, 2));
                // create buffer to read the message
                var payload = new byte[bodyLength];

                // read exact bytes as before
                try
                {
                    await ReadExactBytesAsync(sourceStream, payload, ct);
                }
                catch (OperationCanceledException)
                {
                    throw;
                }
                catch (Exception ex)
                {
                    o.OnError(new InvalidDataException("Error in stream.", ex));
                    return;
                }

                // create a new message and send it to client
                var message = new Message { PayLoad = payload };
                o.OnNext(message);
            }
            // wrap things up
            ct.ThrowIfCancellationRequested();
            o.OnCompleted();
        });

        // return the suscription handle
        dispose.Add(schedule);
        return dispose;
    });
}

EDIT - Very hacky test code I used:

private static void Main(string[] args)
{
    var listener = new TcpListener(IPAddress.Any, 12873);
    listener.Start();

    var listenTask = listener.AcceptTcpClientAsync();
    listenTask.ContinueWith((Task<TcpClient> t) =>
    {
        var client = t.Result;
        var stream = client.GetStream();
        const string messageText = "Hello World!";                
        var body = Encoding.UTF8.GetBytes(messageText);                
        var header = BitConverter.GetBytes(
            IPAddress.HostToNetworkOrder(body.Length));
        for (int i = 0; i < 5; i++)
        {
            stream.Write(header, 0, 4);
            stream.Write(body, 0, 4);
            stream.Flush();
            // deliberate nasty delay
            Thread.Sleep(2000);
            stream.Write(body, 4, body.Length - 4);
            stream.Flush();
        }
        stream.Close();
        listener.Stop();
    });


    var tcpClient = new TcpClient();
    tcpClient.Connect(new IPEndPoint(IPAddress.Loopback, 12873));
    var clientStream = tcpClient.GetStream();

    ReadMessages(clientStream).Subscribe(
        Console.WriteLine,
        ex => Console.WriteLine("Error: " + ex.Message),
        () => Console.WriteLine("Done!"));

    Console.ReadLine();
}

Wrapping up

You need to think about setting a timeout for reads, in case the server dies, and some kind of "end message" should be sent by the server. Currently this method will just continually tries to receive bytes. As you haven't specced it, I haven't included anything like this - but if you do, then as I've written it just breaking out of the while loop will cause OnCompleted to be sent.

Caveat emptor - I've done only minimal testing with this, so use with caution! — James World, Feb 06 '14 at 15:32
Oh - one other point, you could potential simplify the code by using the async flavour of Create instead of ScheduleAsync - but I can't see how to control the Scheduler as easily in that case. Open to suggestions. — James World, Feb 06 '14 at 16:21
To your point about not reading everything in the stream up to the buffer... this is not particularly inefficient - it's not as though the unread bytes are floating in the ether, they *are* still buffered, it happens lower down the stack whatever you do. — James World, Feb 06 '14 at 17:25
Well, this is basically the same solution as the one I linked to, albeit a bit more advanced. I have a solution for that already. The Whole point of the post is really being able to constantly reading whatever is available in the stream and not creating small buffers and read into those all the time. Also trying to learn Rx on the way here. Appreciate the effort you have done though, but it really doesn't answer the question I have. — EMB, Feb 06 '14 at 18:23
Well, just as an observation, producing IObservable makes a stream of lots of extremely small buffers - one byte each! I'll think about it, but I suspect it be nasty code... I would stress that it is a fallacy to think that not reading all available bytes is wasteful - the framing needs to happen somewhere and there is already an underlying buffer. — James World, Feb 06 '14 at 18:34
Do you have particular performance goals in mind? E.g. Reducing GC pressure, achieving a certain number of messages per second,minimising memory use etc... Because at this level of optimization you need clear goals. — James World, Feb 06 '14 at 18:42
First thing that comes to mind is that I could create a Rx enabled Library which is not dependant on the protocol used. I am also Learning here, so trying to wrap my head around Rx at the same time. I am still thinking this can't be that hard, but that is probably because of my Limited knowledge :D — EMB, Feb 06 '14 at 19:19
Reading a Stream and turing it into an Observable sequence is a solved problem. Just choose how you want to push the results back; either a continuous stream of bytes or as packets of bytes. — Lee Campbell, Feb 11 '14 at 18:48

Ahmed Alejo · Answer 2 · 2019-05-10T10:32:23.583

I guess what is needed here is Qactive: A Rx.Net based queryable reactive tcp server provider

Server

Observable
    .Interval(TimeSpan.FromSeconds(1))
    .ServeQbservableTcp(new IPEndPoint(IPAddress.Loopback, 3205))
    .Subscribe();

Client

var datasourceAddress = new IPEndPoint(IPAddress.Loopback, 3205);
var datasource = new TcpQbservableClient<long>(datasourceAddress);

(
     from value in datasource.Query()
     //The code below is actually executed on the server
     where value <= 5 || value >= 8
     select value
)
.Subscribe(Console.WriteLine);

What´s mind blowing about this is that clients can say what and how frequently they want the data they receive and the server can still limit and control when, how frequent and how much data it returns.

For more info on this https://github.com/RxDave/Qactive

Another blog.sample

https://sachabarbs.wordpress.com/2016/12/23/rx-over-the-wire/

Don't forget to call Subscribe on the server-side too! – Dave Sexton May 06 '19 at 11:17 — Dave Sexton, May 06 '19 at 11:17

Read continous bytestream from Stream using TcpClient and Reactive Extensions

2 Answers2

Wrapping up

Linked