Observable.Window and .Zip not functioning like I would expect

Question

I'm trying to turn an IEnumerable into an IObservable that delivers its items in chunks one second apart.

var spartans = Enumerable.Range(0, 300).ToObservable();

spartans
    .Window(30)
    .Zip(Observable.Timer(DateTimeOffset.Now, TimeSpan.FromMilliseconds(1000)), (x, _) => x)
    .SelectMany(w => w)
    .Subscribe(
        n => Console.WriteLine("{0}", n),
        () => Console.WriteLine("all end"));

With this code, the only thing that is printed is "all end" after ten seconds. If I remove the .Zip then the entire sequence prints instantaneously, and if I remove the .Window and .SelectMany then the entire sequence prints one item per second. If I peek into the "windowed" observable inside the lambda passed to SelectMany, I can see that it is empty. My question is, why?

score 2 · Accepted Answer · edited Jun 20 '20 at 09:12

The problem is occurring because of how Window works with a count - and this one isn't particularly intuitive!

As you know, Window serves a stream of streams. However, with a count, the child streams are "warm" - i.e. when an observer of this stream receives a new window in it's OnNext handler, it must subscribe to it before it cedes control back to the observable, or the events are lost.

Zip doesn't "know" it's dealing with this situation, and doesn't give you the opportunity to subscribe to each child window before it grabs the next.

If you remove the Zip, you see all the events because the SelectMany does subscribe to all the child windows as it receives them.

The easiest fix is to use Buffer instead of Window - make that one change and your code works. That's because Buffer works very similarly to SelectMany, effectively preserving the windows by doing this:

Window(30).SelectMany(x => x.ToList())

The elements are no longer warm windows but are crystallized as lists, and your Zip will now work as expected, with the following SelectMany flattening the lists out.

Important Performance Consideration

It's important to note that this approach will cause the entire IEnumerable<T> to be run through in one go. If the source enumerable should be lazily evaluated (which is usually desirable), you'll need to go a different way. Using a downstream observable to control the pace of an upstream one is tricky ground.

Let's replace your enumerable with a helper method so we can see when each batch of 30 is evaluated:

static IEnumerable<int> Spartans()
{
    for(int i = 0; i < 300; i++)
    {
        if(i % 30 == 0)
            Console.WriteLine("30 More!");
        
        yield return i;            
    }
}

And use it like this (with the Buffer "fix" here, but the behaviour is similar with Window):

Spartans().ToObservable()
          .Buffer(30)
          .Zip(Observable.Timer(DateTimeOffset.Now, 
                                TimeSpan.FromMilliseconds(1000)),
               (x, _) => x)
          .SelectMany(w => w)
          .Subscribe(
              n => Console.WriteLine("{0}", n),
              () => Console.WriteLine("all end"));

Then you see this kind of output demonstrating how the source enumerable is drained all at once:

30 More!
0
1
...miss a few...
29
30 More!
30 More!
30 More!
30 More!
30 More!
30 More!
30 More!
30 More!
30 More!
30
31
32
...etc...

To truly pace the source, rather than using ToObservable() directly you could do the following. Note the Buffer operation on the Spartans() IEnumerable<T> comes from nuget package Ix-Main - added by the Rx team to plug a few holes on the IEnumerable<T> monad:

var spartans = Spartans().Buffer(30);
var pace = Observable.Timer(DateTimeOffset.Now, TimeSpan.FromMilliseconds(1000));
       
pace.Zip(spartans, (_,x) => x)
    .SelectMany(x => x)
    .Subscribe(
        n => Console.WriteLine("{0}", n),
        () => Console.WriteLine("all end"));

And the output becomes a probably much more desirable lazily evaluated output:

30 More!
0
1
2
...miss a few...
29
30 More!
30
31
32
...miss a few...
59
30 More!
60
61
62
...etc

Ana Betts · Answer 2 · 2015-02-22T18:20:59.457

0

I'm not sure how to get this working with Window, but what about this:

var spartans = Enumerable.Range(0, 300).ToObservable();

spartans
    .Select(x => Observable.Timer(TimeSpan.FromSeconds(1)).Select(_ => x))
    .Merge(30);

edited Feb 22 '15 at 18:20

answered Feb 22 '15 at 03:37

Ana Betts

73,868
16
141
209

That ends up doing essentially the same thing as the `.Zip`, so one item is emitted every second. I still can't get the `.Window` to work (so that 30 are emitted as a group). – moswald Feb 22 '15 at 04:44
Oh, "In chunks", I totally missed that part - Fixed – Ana Betts Feb 22 '15 at 18:20
This approach still has the effect of draining the whole `IEnumerable` immediately, and the elements (within a chunk) are presented in a non-deterministic order. Of course, neither of those were presented as requirements, but such requirements wouldn't be unusual. – James World Feb 22 '15 at 19:06
I definitely wanted the `IEnumerable` to be lazy, but forgot to mention that part. – moswald Feb 23 '15 at 16:01

Observable.Window and .Zip not functioning like I would expect

2 Answers2

Important Performance Consideration