Combination of Observable.FromAsync+Repeat+TakeWhile creates infinite loop

Question

Can someone explain why the following AsObservable method creates an infinite loop even though the end of stream is reached?

public static class StreamExt {
    public static IObservable<byte> AsObservable(this Stream stream, int bufferSize) {
        return Observable
            .FromAsync(cancel => stream.ReadBytes(bufferSize, cancel))
            .Repeat()
            .TakeWhile(bytes => bytes != null) // EndOfStream
            .SelectMany(bytes => bytes);
    }

    private static async Task<byte[]> ReadBytes(this Stream stream, int bufferSize, CancellationToken cancel) {
        var buf = new byte[bufferSize];
        var bytesRead = await stream
            .ReadAsync(buf, 0, bufferSize, cancel)
            .ConfigureAwait(false);

        if (bytesRead < 1) return null; // EndOfStream
        var result_size = Math.Min(bytesRead, bufferSize);
        Array.Resize(ref buf, result_size);
        return buf;
    }
}

A quick tests shows that it produces an infinite loop:

class Program {
    static void Main(string[] args) {
        using (var stream = new MemoryStream(new byte[] { 1, 2, 3 })) {
            var testResult = stream
                .AsObservable(1024)
                .ToEnumerable()
                .ToArray();
            Console.WriteLine(testResult.Length);
        }
    }
}

Of course I could add an .SubscribeOn(TaskPoolScheduler.Default) but however, the infinite loop stays alive (blocks a task pool scheduler + infinitely reads from Stream).

[UPDATE 2017-05-09]

Shlomo posted a better example to reproduce this issue:

int i = 0;
var testResult = Observable.FromAsync(() => Task.FromResult(i++))
    .Repeat()
    .TakeWhile(l => l < 3);
testResult.Subscribe(b => Console.WriteLine(b), e => { }, () => Console.WriteLine("OnCompleted"));
Console.WriteLine("This is never printed.");

You have a `.Repeat()` call in there, which 'repeats the observable sequence indefinitely`. What is it exactly that you are trying to achieve? — Maarten, May 09 '17 at 09:05
Read the `Stream` until it ends or the user disposes her/his subscription. `TakeWhile` _should_ stop the observable stream if the predicate evaluates with `true` (end of stream). At the moment even `.Dispose()` at the subscription does not stop `.Repeat()`. — Daniel Müller, May 09 '17 at 09:14
Why would `.TakeWhile()` stop if you call it after the `.Repeat()` ? — Maarten, May 09 '17 at 09:18
I do have a solution (using `Observable.Create`). I would like to understand why the combination of `Observable.FromAsync` + `.Repeat()` does not work. — Daniel Müller, May 09 '17 at 09:19
AFAIK it works as it should, since the `.TakeWhile()` has no condition to stop it. `bytes != null` is always true, since you repeat it, and the stream never yields `null`. — Maarten, May 09 '17 at 09:21
It does stop when using e.g. `Observable.Range(1, 50).Repeat().TakeWhile(i => i < 3).Subscribe(Console.WriteLine);` — Daniel Müller, May 09 '17 at 09:23
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/143759/discussion-between-maarten-and-daniel-muller). — Maarten, May 09 '17 at 09:23
No, I've debugged it several times. `bytes != null` evalutes correctly to `false` as soon as the end of stream is reached. But it does not stop the `.Repeat()` loop. — Daniel Müller, May 09 '17 at 09:25
I think it might be related to this https://github.com/Reactive-Extensions/Rx.NET/issues/42 and https://github.com/Reactive-Extensions/Rx.NET/issues/68 . If you add something like .SubscribeOn(TaskPoolScheduler.Default) in between FromAsync and Repeat then it continues on without any issues. — Shane Neuville, May 11 '17 at 01:13
http://stackoverflow.com/questions/8284300/fast-repeat-takewhile-causes-infinite-loop?rq=1 — Shane Neuville, May 11 '17 at 02:55

score 4 · Accepted Answer · answered Oct 23 '19 at 19:32

For anyone who ends up here and needs an answer, not just an explanation: the issue appears to be the default scheduler of FromAsync, as indicated by this self-answered question. If you adjust to the "current thread" scheduler Repeat().TakeWhile(...) behaves more predictably. E.g. (extract from question):

.FromAsync(cancel => stream.ReadBytes(bufferSize, cancel), 
    System.Reactive.Concurrency.Scheduler.CurrentThread)
.Repeat()
.TakeWhile(bytes => bytes != null) // EndOfStream

Shlomo · Answer 2 · 2017-05-09T11:28:11.837

You can confirm the OnCompleted is being produced correctly with this:

using (var stream = new MemoryStream(new byte[] { 1, 2, 3 }))
{
    var testResult = stream
        .AsObservable(1024)
        ;
    testResult.Subscribe(b => Console.WriteLine(b), e => {}, () => Console.WriteLine("OnCompleted"));
}

It looks like there's a problem with the .FromAsync + .Repeat combination. The following code acts similarly:

int i = 0;
var testResult = Observable.FromAsync(() => Task.FromResult(i++))
    .Repeat()
    .TakeWhile(l => l < 3)
    ;
testResult.Subscribe(b => Console.WriteLine(b), e => { }, () => Console.WriteLine("OnCompleted"));
Console.WriteLine("This is never printed.");

...whereas this code terminates correctly:

var testResult = Observable.Generate(0, i => true, i => i + 1, i => i)
    .Repeat()
    .TakeWhile(l => l < 3)
    ;
testResult.Subscribe(b => Console.WriteLine(b), e => { }, () => Console.WriteLine("OnCompleted"));
Console.WriteLine("This is printed.");

score 0 · Answer 3 · answered May 11 '17 at 07:33

0

Shane Neuville posted links that contain an explanation for this behaviour:

answered May 11 '17 at 07:33

Daniel Müller

434
4
12

Combination of Observable.FromAsync+Repeat+TakeWhile creates infinite loop

3 Answers3