How do I poll with state using Reactive Extensions?

Question

There is already a good question on database polling using Reactive (Database polling with Reactive Extensions)

I have a similar question, but with a twist: I need to feed a value from the previous result into the next request. Basically, I would like to poll this:

interface ResultSet<T>
{
   int? CurrentAsOfHandle {get;}
   IList<T> Results {get;}
}

Task<ResultSet<T>> GetNewResultsAsync<T>(int? previousRequestHandle);

The idea is that this returns all new items since the previous request

---

1. every minute I would like to call GetNewResultsAsync
2. I would like to pass the CurrentAsOf from the previous call as the argument to the previousRequest parameter
3. the next call to GetNewResultsAsync should actually happen one minute after the previous one

---

Basically, is there a better way than:

return Observable.Create<IMessage>(async (observer, cancellationToken) =>
{
    int? currentVersion = null;

    while (!cancellationToken.IsCancellationRequested)
    {
        MessageResultSet messageResultSet = await ReadLatestMessagesAsync(currentVersion);

        currentVersion = messageResultSet.CurrentVersionHandle;

        foreach (IMessage resultMessage in messageResultSet.Messages)
            observer.OnNext(resultMessage);

        await Task.Delay(TimeSpan.FromMinutes(1), cancellationToken);
    }
});

Also note that this version allows the messageResultSet to be collected while waiting for the next iteration (for example, I thought maybe I could use Scan to pass the previous result set object into the next iteration)

Answer 1

Your question basically reduces to this: There's a Scan function with the signature:

IObservable<TAccumulate> Scan<TSource, TAccumulate>(this IObservable<TSource> source, 
     TAccumulate initialValue, Func<TAccumulate, TSource, TAccumulate> accumulator)

But you need something like

IObservable<TAccumulate> Scan<TSource, TAccumulate>(this IObservable<TSource> source, 
     TAccumulate initialValue, Func<TAccumulate, TSource, IObservable<TAccumulate>> accumulator)

...where the accumulator function returns an observable, and the Scan function automatically reduces it to pass in to the next call.

Here's a poor man's functional implementation of Scan:

public static IObservable<TAccumulate> MyScan<TSource, TAccumulate>(this IObservable<TSource> source, 
    TAccumulate initialValue, Func<TAccumulate, TSource, TAccumulate> accumulator)
{
    return source
        .Publish(_source => _source
            .Take(1)
            .Select(s => accumulator(initialValue, s))
            .SelectMany(m => _source.MyScan(m, accumulator).StartWith(m))
        );
}

Given that, we can change it a bit to incorporate the reducing functionality:

public static IObservable<TAccumulate> MyObservableScan<TSource, TAccumulate>(this IObservable<TSource> source,
    TAccumulate initialValue, Func<TAccumulate, TSource, IObservable<TAccumulate>> accumulator)
{
    return source
        .Publish(_source => _source
            .Take(1)
            .Select(s => accumulator(initialValue, s))
            .SelectMany(async o => (await o.LastOrDefaultAsync())
                .Let(m => _source
                    .MyObservableScan(m, accumulator)
                    .StartWith(m)
                )
            )
            .Merge()
        );
}

//Wrapper to accommodate easy Task -> Observable transformations
public static IObservable<TAccumulate> MyObservableScan<TSource, TAccumulate>(this IObservable<TSource> source,
    TAccumulate initialValue, Func<TAccumulate, TSource, Task<TAccumulate>> accumulator)
{
    return source.MyObservableScan(initialValue, (a, s) => Observable.FromAsync(() => accumulator(a, s)));  
}

//Function to prevent re-evaluation in functional scenarios
public static U Let<T, U>(this T t, Func<T, U> selector)
{
    return selector(t);
}

Now that we have this fancy MyObservableScan operator, we can solve your problem relatively easily:

var o = Observable.Interval(TimeSpan.FromMinutes(1))
    .MyObservableScan<long, ResultSet<string>>(null, (r, _) => Methods.GetNewResultsAsync<string>(r?.CurrentAsOfHandle))

Please note that in testing I have noticed that if the accumulator Task/Observable function takes longer than intervals in the source, the observable will terminate. I'm not sure why. If someone can correct, much obliged.

Answer 2

I since found that there is an overload to Observable.Generate that pretty much does the trick. The main drawback is that it doesn't work with async.

public static IObservable<TResult> Generate<TState, TResult>(TState initialState, Func<TState, bool> condition, Func<TState, TState> iterate, Func<TState, TResult> resultSelector, Func<TState, TimeSpan> timeSelector, IScheduler scheduler);

I pass in null as my initial state. Pass in x => true as my condition (to poll endlessly). Inside of the iterate, I do the actual polling, based on the state that was passed in. Then in timeSelector I return the polling interval.

So:

var resultSets = Observable.Generate<ResultSet<IMessage>, IEnumerable<IMessage>>(
   //initial (empty) result
   new ResultSet<IMessage>(),

   //poll endlessly (until unsubscription)
   rs => true,

   //each polling iteration
   rs => 
   {
      //get the version from the previous result (which could be that initial result)
      int? previousVersion = rs.CurrentVersionHandle;

      //here's the problem, though: it won't work with async methods :(
      MessageResultSet messageResultSet = ReadLatestMessagesAsync(currentVersion).Result;

      return messageResultSet;
   },

   //we only care about spitting out the messages in a result set
   rs => rs.Messages, 

   //polling interval
   TimeSpan.FromMinutes(1),

   //which scheduler to run each iteration 
   TaskPoolScheduler.Default);

return resultSets
  //project the list of messages into a sequence
  .SelectMany(messageList => messageList);