How to concat async enumerables?

Question

I have a method with this return type:

public async Task<IEnumerable<T>> GetAll()

It makes some further async calls (unknown number) each of which return a task of enumerable T, and then wants to concat the results for the return.

var data1 = src1.GetAll();
var data2 = src2.GetAll();
var data3 = src3.GetAll(); //and so on

Now it's easy enough to await all and concat the results to produce the single enumerable, but i'd like the enumerable to be available as soon as the first call returns, with potential waits for the caller / enumerator if any calls are still pending when available results run out.

Do i have to hand-roll a concat for this, working around the lack of enumerator support when it's wrapped in a task<>? Or there's already a library call in TPL or elsewhere which could help me. I did look at IX, but it's still on experimental release and don't want to fold it in.

On a side note, is what i'm trying an anti-pattern? I can think of one complication, exception handling - from the caller's side, the call can complete successfully and he starts using the enumerable but it can blow up midway through that...

(As a side note) Your pattern seems to be asymmetric: the code _waits for_ the first sequence to be ready, but doesn't wait for the rest. — Vlad, Feb 23 '15 at 16:16
Maybe you really need `IObservable`? In that case you could just use [`Observable.Concat`](https://msdn.microsoft.com/en-us/library/system.reactive.linq.observable.concat%28v=vs.103%29.aspx). — Vlad, Feb 23 '15 at 16:18
@Vlad:Good point. I did think of it when i looked into Ix / Rx and i agree, reactive does tend to fit this type of use case better. But the overall scenario and application is a pull based one and i don't want to drop in the Rx hammer just to go back to enumerables from the observable result. — Vivek, Feb 23 '15 at 17:45

Yuval Itzchakov · Accepted Answer · 2015-02-23T16:23:15.633

There is an existing project called Async Enumerable which answers this problem exactly.

You could put it to use quite easily.

For example:

IAsyncEnumerable<string> GetAsyncAnswers()
{
    return AsyncEnum.Enumerate<string>(async consumer =>
    {
        foreach (var question in GetQuestions())
        {
            string theAnswer = await answeringService.GetAnswer(question);
            await consumer.YieldAsync(theAnswer);
        }
    });
}

This exposes an IAsyncEnumerable<string> which yields once GetAnswer returns. You could internally expose an IAsyncEnumerable<T> in your case and internally make calls inside GetAll.

what i'm trying an anti-pattern? I can think of one complication, exception handling - from the caller's side, the call can complete successfully and he starts using the enumerable but it can blow up midway through that...

I wouldn't say so. This does have potentially problems such as an exception occurring internally during one of the awaits, but this could also potentially happen inside any IEnumerable<T>. Asynchronous sequences are something needed in todays reality of emerging async API's.

Thanks. Marked as answer. I didn't import the library, but ended by rolling my own anyways in a similar two lines to your sample, via a concatenator taking a func for getting the next enumerable. Maybe i'll take another look at the library if i get a broader use case. — Vivek, Feb 23 '15 at 23:34

How to concat async enumerables?

1 Answers1