Why does this Observable.Generate overload cause a memory leak? [Using Timespan < 15ms]

Question

The following Rx.NET code will use up about 500 MB of memory after about 10 seconds on my machine.

var stream =
    Observable.Range(0, 10000)
              .SelectMany(i => Observable.Generate(
                  0, 
                  j => true, 
                  j => j + 1, 
                  j => new { N = j },
                  j => TimeSpan.FromMilliseconds(1)));

stream.Subscribe();

If I use the Observable.Generate overload without a Func<int, TimeSpan> parameter my memory usage plateaus at 35 MB.

var stream =
    Observable.Range(0, 10000)
              .SelectMany(i => Observable.Generate(
                  0,
                  j => true,
                  j => j + 1,
                  j => new { N = j }));
                  // j => TimeSpan.FromMilliseconds(1))); ** Removed! **

stream.Subscribe();

It seems to only be a problem when using SelectMany() or Merge() extension methods.

See http://stackoverflow.com/questions/41223723/observable-generate-with-timespan-selector-appears-to-leak-memory-when-using-a?noredirect=1&lq=1 for an explanation of why I added the TimeSpan qualification to the question title. — James World, Dec 20 '16 at 14:21

Enigmativity · Accepted Answer · 2015-09-18T23:21:34.603

This is an issue of which default scheduler is used.

With the TimeSpan version the scheduler is the DefaultScheduler. Without TimeSpan it is CurrentThreadScheduler.

So, for the time-based generate it's very rapidly trying to schedule all of the operations and basically builds up a massive queue of events waiting to be executed. Thus it uses a load of memory.

With the non-time-based generate it's using the current thread so it will produce and consume each generated value in series and thus use very little memory.

Oh, and this isn't a memory leak. It's just the normal operation if you try to schedule an infinite number of values faster than they can be consumed.

I decompiled the code to work out which schedulers were used.

Here's the non-time-based decompile:

public static IObservable<TResult> Generate<TState, TResult>(TState initialState, Func<TState, bool> condition, Func<TState, TState> iterate, Func<TState, TResult> resultSelector)
{
    if (condition == null)
        throw new ArgumentNullException("condition");
    if (iterate == null)
        throw new ArgumentNullException("iterate");
    if (resultSelector == null)
        throw new ArgumentNullException("resultSelector");
    return Observable.s_impl.Generate<TState, TResult>(initialState, condition, iterate, resultSelector);
}

public virtual IObservable<TResult> Generate<TState, TResult>(TState initialState, Func<TState, bool> condition, Func<TState, TState> iterate, Func<TState, TResult> resultSelector)
{
    return (IObservable<TResult>)new Generate<TState, TResult>(initialState, condition, iterate, resultSelector, SchedulerDefaults.Iteration);
}

internal static IScheduler Iteration
{
    get
    {
        return (IScheduler)CurrentThreadScheduler.Instance;
    }
}

The above methods are from Observable, QueryLanguage, and SchedulerDefaults respectively.

I tried the `Observable.Generate` overload with the timeSelector and scheduler parameters with different schedulers. The `CurrentThreadScheduler` from `Scheduler.CurrentThread` still builds up a queue of events until OoM exception. The `ImmediateScheduler` from `Scheduler.Immediate` seems to make the stream operate like the `Observable.Generate` without the `TimeSpan`. Are you sure the default scheduler is `CurrentThreadScheduler` and not `ImmediateScheduler`? — voqk, Sep 18 '15 at 15:11
@voqk - Yes, I am. Please see the decompile I added to the answer. — Enigmativity, Sep 18 '15 at 23:21

Why does this Observable.Generate overload cause a memory leak? [Using Timespan < 15ms]

1 Answers1

Linked