Signal Filter with Rx.NET

Question

I'd like to implement a signal filter in Rx.NET which starts with an initial set of coefficients. As time goes by the filter coefficients have to be recalculated from a snapshot of observed data values.

Here is a small prototype which shows how it should work. For simplicity I choose the filter length and the number of the historical data values used to recalculate the filter coefficients to be the same (3 in the example).

The example uses a side effect in bufferedAt10 to recalculate the coefficients. This is not what I would like to have.

In the real application the data is coming in irregular time steps and the coefficient should be updated once a day or once a week at a specific time. I can easily make the buffer longer, but how can I let the system run and change the filter coefficients from a observer in a clean functional way?

       // create a hot obvervable to produce data
        const int bufLen = 3;
        var rng = new Random();
        var period = TimeSpan.FromSeconds(0.5);
        var observable = Observable.Interval(period)
        .Select(i => new {Time = DateTime.Now, Price = rng.NextDouble()})
        .Do(e => Console.WriteLine("original : {0}", e))
        .Publish();
        observable.Connect();

        Console.WriteLine("Press any key to subscribe");
        Console.ReadKey();

        // buffer of length bufLen used for filter calculation (every tick) and filter 
        // coefficient update (at a lower frequency)
        var buffered = observable.Buffer(bufLen, 1);

        // apply the signal filter with coefficients in `coeff`
        var coeff = new List<Double>() {1.0, 1.0, 1.0};  // these will be updated on the way from new data 
        var filtered = buffered.Select(e =>
        {
            var f = 0.0;
            for (var i = 0; i < bufLen; i++)
            {
                f += e[i].Price*coeff[i]; // apply the filter with coefficients `coeff`
            }
            return new {Time = DateTime.Now, FilteredPrice = f};
        });

        var s1 = filtered.Subscribe(e => Console.WriteLine("filtered : {0} (coeff {1},{2},{3})", e, coeff[0], coeff[1], coeff[2]));

        // recalculate the filter coefficients say every 10 seconds 
        var bufferedAt10 = buffered.DistinctUntilChanged(e => (e[bufLen - 1].Time.TimeOfDay.Seconds / 10) * 10);

        var s2 = bufferedAt10.Subscribe(e =>
        {
            Console.WriteLine("recalc of coeff : {0}", e[bufLen - 1].Time);
            for (var i = 0; i < bufLen; i++)
            {
                // a prototypical function that takes the buffer and uses it to "recalibrate" the filter coefficients
                coeff[i] = coeff[i] + e[bufLen - 1 - i].Price;
            }
            Console.WriteLine("updated coeffs to {0},{1},{2}", coeff[0], coeff[1], coeff[2]);
        });

Thanks for any good advice.

Answer 1

The following is untested but I think it should cover what you need. The idea behind it is that you diverge the streams do the coefficient updates on one, and then bring them back together with WithLatestFrom. I used Sample and Scan to perform the period "adjustment". And your custom timestamping can be done by using the TimeStamp operator instead. You may also consider moving the Publish down past Buffer otherwise you will have two streams generating buffers, but it is up to you.

const int bufLen = 3;
var rng = new Random();
var period = TimeSpan.FromSeconds(0.5);
var observable = Observable.Interval(period)
  .Select( => rng.NextDouble())
  .Publish();
observable.Connect();

Console.WriteLine("Press any key to subscribe");
Console.ReadKey();

var buffered = observable.Buffer(bufLen, 1);

var seed = new [] {1.0, 1.0, 1.0};

var coefficients = buffered
                     //Samples for a new value every 10 seconds
                     .Sample(TimeSpan.FromSeconds(10))
                     //Updates the seed value and emits it after every update
                     .Scan(seed, 
                       //Use good old fashion Linq
                       (coeff, delta) => coeff.Zip(delta.Reverse(), 
                                         (c, d) => c + d.Price)
                                         .ToArray()
                       );

//Emits a new value everytime buffer emits, and combines it with the latest
//values from the coefficients Observable
//Kick off coefficients with the seed otherwise you need to wait 10 seconds 
//for the first value.
buffer.WithLatestFrom(coefficients.StartWith(seed), (e, coeff) => {
  return e.Zip(coeff, (x, c) => x.Price * c).Sum();
})
.TimeStamp()
.Subscribe(e => Console.WriteLine("filtered : {0}", e);