-1

An application (Saver) receives live data over a websocket from a remote server and stores it in a database. It exposes a REST endpoint to clients that returns all data stored in the database so far.

A client application subscribes to live data on the remote server's websocket on start up. It then makes the request to Saver's REST endpoint, and receives all data so far.

Both data sources are exposed as IObservable<AType> in the client application.

AType includes a Timestamp property.

How can I combine these two Observables so that they are sequential (by timestamp) without duplicates?

Update: duplicates are not possible within either one of the data sources/Observables, but are possible when they are combined, since the websocket is subscribed to before the REST endpoint is called. They are subscribed to in that order to avoid loss of data.

arbitrary
  • 103
  • 9

2 Answers2

2

Ordering a real time stream by a value without buffering/windowing is not possible. You need to explicitly buffer the merged streams (Observable.Merge) using Observable.Buffer/Observable.Window operators and then you can use Enumerable.SortByto sort by timestamp.

Here is the relevant issue in the rxnet github page: https://github.com/Reactive-Extensions/Rx.NET/issues/122 .

There is a Observable.Distinct operator but be careful using it on long running /high throughput streams as it stores the hashes of the values to detect duplicates.

ekim boran
  • 1,809
  • 12
  • 22
0

My working solution:

Subscribe to websocket

Retrieve all data so far over REST

If anything has arrived over websocket, get the timestamp of the first item and remove this and anything later from the REST results

Combine the two streams like this:

        combinedObservable = truncatedRestData.ToObservable().Concat(socketObservable);
arbitrary
  • 103
  • 9