Consume a StreamReader stream .NET from a web request as it streams?

Question

I'm experimenting with the Twitter streaming api, and am trying to open a stream for a user to consume events as they happen. I'm using a standard set of classes for making REST api calls to twitter. When using https://userstream.twitter.com/2/user.json in a "GET" call the response stream never ends... I'm opening a StreamReader and reading the response as I would with any other REST call to Twitter. This is probably obvious to others, but how do I "consume" this stream... Is there a way to read the StreamReader as it's reading (meaning before it closes)? or maybe there a different method I can user to consume this stream.... again, I apologize if this seams to be elementary to some, but I can't figure it out at the moment... Thanks in advance for any advise or help!!!

Here is the original source I started troubleshooting this with... This method was fabricated from a set of C# Classes I found in a forum on LinkedIn. At the line that reads "responseData = responseReader.ReadToEnd()" the method starts to "drink" the stream... but does so like a bottomless cup... reading this stream of data in real time before it closes (which is essentially until I stop debugging or kill the process) is the question I'm tackling.

Private Function WebResponseGet(ByVal webRequest As HttpWebRequest) As String
    Dim responseReader As StreamReader = Nothing
    Dim responseData As String = ""

    Try
        responseReader = New StreamReader(webRequest.GetResponse().GetResponseStream())
        responseData = responseReader.ReadToEnd()
    Catch
        Throw
    Finally
        webRequest.GetResponse().GetResponseStream().Close()
        responseReader.Close()
        responseReader = Nothing
    End Try

    Return responseData
End Function

UPDATE & RELATED QUESTION:

So, I figured out the following way to keep a stream open, and write it to a file (this won't be the final approach, I'm just testing, a developing the best way of doing this :)

Private Sub DrinkIt(ByVal webRequest As HttpWebRequest)

    Dim coder As Encoding = Encoding.UTF8

    Dim stream_reader As New StreamReader(webRequest.GetResponse().GetResponseStream(), coder, True, 1024)

    Do While 0 < 1

        Dim w As IO.StreamWriter

        w = File.AppendText(targetFile)

        Dim c(5) As Char

        stream_reader.Read(c, 0, c.Length)

        w.Write(c)

        w.Close()

        w.Dispose()

    Loop

    stream_reader.Close()

    stream_reader.DiscardBufferedData()

    stream_reader.Dispose()

End Sub

This writes the opened twitter stream to a file, and every time I Tweet, Retweet, Delete, Direct Message... and so on... The file grows with a JSON objects appended to the text. I used the Do While 0 < 1 for testing here, because I just wanted to see it working. I see on MSDN StreamReader Constructor Description that the New constructor is supposed to accept a Boolean value for "leaveOpen", but no such argument allowed when I try to add this to the constructor... Does anyone have a good example of how to do this with forcing and infinite loop or just a better approach than this... I would like to simply read new updates sent each time from Twitter, and address them accordingly? There is obviously a way, I'm just new to the concept of consuming a stream like this with out it being closed.. (**btw, Thanks to Dr. Evil's suggestion, I was lead in this direction... It's not exactly what he suggested, but is what lead me here)

please show some source... what have you tried ? what is the goal/what is not working ? — Yahia, Nov 12 '11 at 00:24
Yahia, I have updated the ? with my original source and described the goal... essentially to read this stream, and based on this stream execute other blocks of code. — wakurth, Nov 12 '11 at 05:59
Please see the links in my answer - you will find source code and explanation on how to deal with the Twitter streaming API... — Yahia, Nov 12 '11 at 11:39
Make sure whenever dealing with streams (Any disposable objects really) to always surround them with the Using statement: http://msdn.microsoft.com/en-us/library/htd05whh(v=vs.100).aspx — Mataniko, Jun 20 '13 at 22:00

score 2 · Answer 1 · answered Nov 12 '11 at 00:42

2

Use StreamReader's buffer to feed received bytes to (Encoding.UTF8.GetString) then append them to a StringBuilder to get received the stream as string.

Use ToString of StringBuilder

answered Nov 12 '11 at 00:42

dr. evil

26,944
33
131
201

Hello Dr. Evil ... I am testing this, and will post an edit to this question with my code based on your suggestion in due time. Thank you for the insight sir. – wakurth Nov 12 '11 at 05:53

Yahia · Accepted Answer · 2011-11-12T16:23:28.687

1

One remark upfront: The leaveOpen parameter you saw is only present in .NET 4.5 and up

As to how to deal with the Twitter streaming API:

The above links lead you to opensource libraries and documentation on how to access the Twitter API (both the streaming API and the REST API) with .NET / C#.

edited Nov 12 '11 at 16:23

answered Nov 12 '11 at 11:38

Yahia

69,653
9
115
144

Thank you Yahia! This is perfect. Good point, about the "LeaveOpen" parameter, I was using 4.0... I checked in VS 2011 dev release, and it's there. Also, the cool thing is I already created my own REST TweetDotNet class library, that works mint... I just have to make a specific streaming class for this and inherit the core OAuth methods.. Adding message to a queue is brilliant, I have no idea why I didn't think of that... the top link answered the question specifically, and the other links help in adapting a solution from scratch. – wakurth Nov 12 '11 at 19:12

Consume a StreamReader stream .NET from a web request as it streams?

2 Answers2

Linked