2

Currently I'm working on an app that reads the stream from Twitter API and parses it into objects. At the moment I read the stream and use ReadObject(...) from DataContractJsonSerializer to make my objects and I write them to a buffer in memory (don't worry I read them from that buffer asynchronously and I keep a maximum of 100 objects before I start overwriting old ones).

This works great!! HOWEVER: Do I have the guarantee that the reading/writing will keep up with the actual stream. If this is not the case; what can I do about this?

tshepang
  • 12,111
  • 21
  • 91
  • 136
Jasper
  • 129
  • 1
  • 10
  • 1
    Posting some code always helps us understand better what you are asking. – Tomislav Markovski Jan 16 '12 at 15:01
  • Possible duplicate of [How do I keep reading an endless stream with fail tolerance](http://stackoverflow.com/questions/8866812/how-do-i-keep-reading-an-endless-stream-with-fail-tolerance) – vgru Jan 16 '12 at 15:39

2 Answers2

1

You could use a BlockingCollection for the buffer, that way instead of overwriting old entries, an attempt to add more than 100 items will block instead while your reader catches up.

millimoose
  • 39,073
  • 9
  • 82
  • 134
0

From what I understand, you will not have that guarantee. If you've got a limit of 100 buffered objects, you may get to the point where that buffer is full of new objects, a new one comes in and overwrites something. Really it's a trade off, the more you allow in your buffer the greater security of not falling behind, versus using more RAM.

The only alternative I can see is somehow writing your own scheduler proritising the processing of the buffered objects over reading new ones from the stream.

Alexander R
  • 2,468
  • 24
  • 28
  • Well, in case the deserializing does not keep up: how can i stop reading where i am and continue more downsteam? – Jasper Jan 16 '12 at 21:30
  • The `BlockingCollection` mentioned above may do the trick there. It's either something like that, or actually writing your own thread scheduler or something. I'm guessing you probably don't want to be doing that. – Alexander R Jan 17 '12 at 11:44