How do I receive live data from a long lived HTTP connection in C#?

Question

I'm looking to process the results of a long-lived HTTP connection from a server I am integrating with as they happen. This server returns one line of JSON (\n delimited) per "event" I wish to process.

Given an instance of Stream assigned to the variable changeStream that represents bytes from the open HTTP connection, here's an extracted example of what I'm doing:

(request is an instance of WebRequest, configured to open a connection to the server I am integrating with.)

var response = request.GetResponse();
var changeStream = response.GetResponseStream();

var lineByLine = Observable.Using(
    () => new StreamReader(changeStream),
    (streamReader) => streamReader.ReadLineAsync().ToObservable()
);

lineByLine.Subscribe((string line) =>
{
    System.Console.WriteLine($"JSON! ---------=> {line}");
});

Using the code above, what ends up happening is I receive the first line that the server sends, but then none after that. Neither ones from the initial response, nor new ones generated by real time activity.

For the purposes of my question, assume this connection will remain open indefinitely.
How do I go about having system.reactive trigger a callback for each line as I encounter them?

Please note: This scenario is not a candidate for switching to SignalR

score 2 · Accepted Answer · answered Mar 25 '20 at 04:18

2

Even though this looks more complicated, it is better to use the built-in operators to make this work.

IObservable<string> query =
    Observable
        .FromAsync(() => request.GetResponseAsync())
        .SelectMany(
            r => Observable.Using(
                () => r,
                r2 => Observable.Using(
                    () => r2.GetResponseStream(),
                    rs => Observable.Using(
                        () => new StreamReader(rs),
                        sr =>
                            Observable
                                .Defer(() => Observable.Start(() => sr.ReadLine()))
                                .Repeat()
                                .TakeWhile(w => w != null)))));

It's untested, but it should be close.

answered Mar 25 '20 at 04:18

Enigmativity

113,464
11
89
172

Ooh, I like this. I'll likely remove the `null` cancel as that's not necessary for my scenario. How could I handle reconnects if the connection ever drops? – Alexander Trauzzi Mar 26 '20 at 00:57
1

@AlexanderTrauzzi - You'd need to include the code that does the connection and gets the request. It's just the same as above - just more nesting. Then you can wrap that in a `.Defer` and end with a `.Retry`. – Enigmativity Mar 26 '20 at 01:06
Also, removing the `.TakeWhile(w => w != null)` means that this observable would never naturally end. You'd have to make sure you dispose of any subscriptions manually. – Enigmativity Mar 26 '20 at 01:07
If the disconnection is noticed by an exception when trying to read, is there any special handling needed? – Alexander Trauzzi Mar 26 '20 at 01:11
What I'm noticing now is that when I do get null, it goes into an infinite loop of reading that same null. How would I have it call `sr.DiscardBufferedData()` on `null` and then continuing to read? – Alexander Trauzzi Mar 26 '20 at 01:43
1

@AlexanderTrauzzi - That's what the `.Defer` and `.Retry` to - basically bypass exceptions. – Enigmativity Mar 26 '20 at 02:13
1

@AlexanderTrauzzi - To call `sr.DiscardBufferedData()` you could put a `.Finally` call after the `TakeWhile`. – Enigmativity Mar 26 '20 at 02:15
Okay, so final question -- how do I get access to the `StreamReader` in a `Finally()`? All my `Catches` and `Finallys` only end up receiving a `string`. – Alexander Trauzzi Mar 26 '20 at 02:35
1

@AlexanderTrauzzi - The `.Finally` would be immediately after the `.TakeWhile(w => w != null)`. That's where the `sr` is still in scope. – Enigmativity Mar 26 '20 at 05:02
1

@AlexanderTrauzzi - Yes, but the `sr` is still in scope there too from the outer level. – Enigmativity Mar 26 '20 at 10:51
Right! Thank you :) – Alexander Trauzzi Mar 26 '20 at 10:59
Take a look at let me know what you think! -- https://gist.github.com/atrauzzi/9259f20ea8b97b22c540caf1d0eb4b9a – Alexander Trauzzi Mar 26 '20 at 23:04
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/210407/discussion-between-enigmativity-and-alexander-trauzzi). – Enigmativity Mar 26 '20 at 23:41

score 1 · Answer 2 · answered Mar 23 '20 at 01:37

1

Your attempt will only observe a single ReadLineAsync call. Instead you need to return each line. Probably something like this;

Observable.Create<string>(async o => {
    var response = await request.GetResponseAsync();
    var changeStream = response.GetResponseStream();
    using var streamReader = new StreamReader(changeStream);
    while (true)
    {
        var line = await streamReader.ReadLineAsync();
        if (line == null)
            break;
        o.OnNext(line);
    }
    o.OnCompleted();
});

answered Mar 23 '20 at 01:37

Jeremy Lakeman

9,515
25
29

Is there any way this can be adapted to support fetching lines indefinitely? The connection is long lived and will be adding lines over time. – Alexander Trauzzi Mar 23 '20 at 03:01
1

So long as the server is using chunked encoding, that should be fine. You should probably add some form of retry / reconnect handling to the loop as well... – Jeremy Lakeman Mar 23 '20 at 03:13
There's a whole bunch of `IDisposable`s here are aren't being disposed. It looks like a fairly simple solution, but it's faulty because of two things - you need to dispose - and the observable completes synchronously with the subscription. – Enigmativity Mar 25 '20 at 04:22

How do I receive live data from a long lived HTTP connection in C#?

2 Answers2