IAsyncEnumerable and throttling

Question

Is it possible to throttle the data when consuming IAsyncEnumerable<T>? I have a stream of data coming in rapidly, and I'm only interested in the last element every N seconds.

Thanks.

The consumer drives the loop for fetching more elements from an `IAsyncEnumerable`. But the specific implementation might have already read ahead. But do you mean that you are fetching a new IAsyncEnumerable every N seconds, or waiting N seconds between enumerating individual elements? — Jeremy Lakeman, Jun 16 '22 at 05:04
This is a strange question. The enumerables don't have the concept of "coming fast". They are not observables. You can only talk about the delay they impose when you enumerate them. What does this `IAsyncEnumerable` represent? Where is it coming from? Could you tell us more about it? — Theodor Zoulias, Jun 16 '22 at 11:26

score 0 · Answer 1 · answered Jun 16 '22 at 04:32

It would seem to make sense to have a different endpoint which returns the most recent event rather than a dealing with the stream.

If you have to deal with a queue/stream you could consume the events and assign each new incoming one to something like latest and read that at the desired interval.

score 0 · Answer 2 · answered Jun 16 '22 at 04:53

One solution is to convert IAsyncEnumerable<T> to IObservable<T> and leverage power of System.Reactive

First, you need a converter. I couldn't find builtin so I've created my own

using System;
using System.Collections.Generic;
using System.Reactive.Subjects;

public class AsyncEnumerableToObservable<T> : IObservable<T>
{
    private readonly IAsyncEnumerable<T> _source;
    private readonly Subject<T> _subject = new();

    public AsyncEnumerableToObservable(IAsyncEnumerable<T> source)
    {
        _source = source;
        BeginConsume();
    }

    public IDisposable Subscribe(IObserver<T> observer) => _subject.Subscribe(observer);

    private async void BeginConsume()
    {
        try
        {
            await foreach (var item in _source)
            {
                _subject.OnNext(item);
            }

            _subject.OnCompleted();
        }
        catch (Exception e)
        {
            _subject.OnError(e);
        }
    }
}

public static class AsyncEnumerableExtensions
{
    public static IObservable<T> ToObservable<T>(this IAsyncEnumerable<T> source)
    {
        return new AsyncEnumerableToObservable<T>(source);
    }
}

With this converter you can use myEnumerable.ToObservable() and use Sample method for throttling from System.Reactive

static class Program
{
    static async Task Main(string[] args)
    {
        IAsyncEnumerable<int> seq = CreateSeq();
        seq.ToObservable().Sample(TimeSpan.FromSeconds(1)).Subscribe(Console.WriteLine);

        await Task.Delay(TimeSpan.FromSeconds(10));
    }

    private static async IAsyncEnumerable<int> CreateSeq()
    {
        int i = 0;
        while (true)
        {
            await Task.Yield();
            i++;
            yield return i;
        }
    }
}

There is a built-in `ToObservable` operator in the [System.Linq.Async](https://www.nuget.org/packages/System.Linq.Async/) package. Beware of conversions between enumerable and observable sequences. Enumerable sequences are meant to be enumerated. They produce the next item only when the consumer requests it (`MoveNextAsync`). On the other hand observable sequences produce items autonomously. They don't negotiate with the consumers. So converting from the one model to the other might have unexpected consequences. — Theodor Zoulias, Jun 16 '22 at 11:16

IAsyncEnumerable and throttling

2 Answers2