Processing a Stream into IAsyncEnumerable - Stream is not readable

Question

I have a workflow where I try to do the following:

• A method accepting a callback, which internally produces a Stream and the callers of that method can use the callback to process the Stream whichever way they want
• In one particular case, the caller is using the callback to produce an IAsyncEnumerable from the Stream.

I have created a minimal reproducing example below:

class Program
{
    private static async Task<Stream> GetStream()
    {
        var text =
            @"Multi-line
            string";

        await Task.Yield();

        var bytes = Encoding.UTF8.GetBytes(text);
        return new MemoryStream(bytes);
    }

    private static async Task<T> StreamData<T>(Func<Stream, T> streamAction)
    {
        await using var stream = await GetStream();
        return streamAction(stream);
    }

    private static async Task StreamData(Func<Stream, Task> streamAction)
    {
        await using var stream = await GetStream();
        await streamAction(stream);
    }

    private static async IAsyncEnumerable<string> GetTextLinesFromStream(Stream stream)
    {
        using var reader = new StreamReader(stream);

        var line = await reader.ReadLineAsync();
        while (line != null)
        {
            yield return line;
            line = await reader.ReadLineAsync();
        }
    }

    private static async Task Test1()
    {
        async Task GetRecords(Stream str)
        {
            await foreach(var line in GetTextLinesFromStream(str))
                Console.WriteLine(line);
        }

        await StreamData(GetRecords);
    }

    private static async Task Test2()
    {
        await foreach(var line in await StreamData(GetTextLinesFromStream))
            Console.WriteLine(line);
    }

    static async Task Main(string[] args)
    {
        await Test1();
        await Test2();
    }
}  

Here, method Test1 works fine, whereas Test2 does not, failing with Stream is not readable. The problem is that in the second case when the code gets to processing the actual stream, the stream is already disposed.

Presumably the difference between the two examples is that for the first one, reading the stream is performed while still in the context of the disposable stream, whereas in the second case we are already out.

However, I would argue the second case could be valid too - at least I feel it's quite C#-idiomatic. Is there anything I am missing to get the second case to work too?

Answer 1

The problem with the Test2 approach is that the Stream is disposed when the IAsyncEnumerable<string> is created, not when its enumeration has completed.

The Test2 method uses the first StreamData overload, the one that returns a Task<T>. The T in this case is an IAsyncEnumerable<string>. So the StreamData method returns a task that produces an async-sequence, and then immediately disposes the stream (after producing the sequence). Obviously this is not the right moment to dispose the stream. The right moment would be after the await foreach loop has completed.

In order to make the Test2 work transparently, you should add a third overload of the StreamData method that returns a Task<IAsyncEnumerable<T>> (instead of Task or Task<T>). This overload should return a specialized asynchronous sequence that is tied to a disposable resource, and disposes this resource when its enumeration has completed. Below is an implementation of such a sequence:

public class AsyncEnumerableDisposable<T> : IAsyncEnumerable<T>
{
    private readonly IAsyncEnumerable<T> _source;
    private readonly IAsyncDisposable _disposable;

    public AsyncEnumerableDisposable(IAsyncEnumerable<T> source,
        IAsyncDisposable disposable)
    {
        // Arguments validation omitted
        _source = source;
        _disposable = disposable;
    }

    async IAsyncEnumerator<T> IAsyncEnumerable<T>.GetAsyncEnumerator(
        CancellationToken cancellationToken)
    {
        await using (_disposable.ConfigureAwait(false))
            await foreach (var item in _source
                .WithCancellation(cancellationToken)
                .ConfigureAwait(false)) yield return item;
    }
}

You could use it in the StreamData method like this:

private static async Task<IAsyncEnumerable<T>> StreamData<T>(
    Func<Stream, IAsyncEnumerable<T>> streamAction)
{
    var stream = await GetStream();
    return new AsyncEnumerableDisposable<T>(streamAction(stream), stream);
}

Bear in mind that in general an IAsyncEnumerable<T> can be enumerated multiple times during its lifetime, and by wrapping it into an AsyncEnumerableDisposable<T> it is essentially reduced to a single-enumeration sequence (because the resource will be disposed after the first enumeration).

---

Alternative: The System.Interactive.Async package contains the AsyncEnumerableEx.Using operator, that can be used instead of the custom AsyncEnumerableDisposable class:

private static async Task<IAsyncEnumerable<T>> StreamData<T>(
    Func<Stream, IAsyncEnumerable<T>> streamAction)
{
    var stream = await GetStream();
    return AsyncEnumerableEx.Using(() => stream, streamAction);
}

The difference is that the Stream will be disposed synchronously, via its Dispose method. AFAICS there is no support for disposing IAsyncDisposables in this package.

Here is the signature of the AsyncEnumerableEx.Using method:

// Constructs an async-enumerable sequence that depends on a resource object, whose
// lifetime is tied to the resulting async-enumerable sequence's lifetime.
public static IAsyncEnumerable<TSource> Using<TSource, TResource>(
    Func<TResource> resourceFactory,
    Func<TResource, IAsyncEnumerable<TSource>> enumerableFactory)
    where TResource : IDisposable;