How to await on IObservable of IObservables?

Question

I have an IObservable of objects, which I want to convert to dictionary of lists asynchronously.

Here is my code, where GetSource returns an IObservable:

await GetSource(...)
  .GroupBy(o => o.SiteId)
  .ToDictionary(g => g.Key, g => g.ToList())

Of course, it is wrong, because the result is IDictionary<int, IObservable<IList<T>>>, but I need an IDictionary<int, IList<T>>

In essence, I need to await each and every IObservable<IList<T>>, but I do not know how to do it as elegantly as I am convinced is possible with Rx.NET.

Any ideas?

James World · Accepted Answer · 2015-03-10T19:00:32.170

Note that whatever you do, you can never complete the dictionary until after the source has completed. A simple approach would therefore be to asynchronously obtain the list like this:

await GetSource(...).ToList();

and then proceed with the resulting IList<T> as you were with the GroupBy and ToDictionary - noting that you are now using the IEnumerable<T> implementations of these operators against a completed list.

This approach assumes that amount and timing of the events in the GetSource() observable, and the distribution between the groups is such that the expense of computing the groups and dictionaries on the completed stream is not prohibitive. Since ultimately you are returning a completed dictionary which will be held in memory anyway we are likely only considering the cost of sorting into groups and creating dictionary entries. This will be extremely fast on in memory data compared to the serialized delivery of a lengthy event stream, so I tend to think that there are few cases where this approach isn't perfectly fine.

If it is prohibitive, it may be worth building up each group and list concurrently and the events arrive, in which case you could do this instead:

await GetSource(...).GroupBy(x => x.SiteId)
                    .SelectMany(x => x.ToList())
                    .ToDictionary(x => x[0].SiteId);

Note that the first element of the list in the ToDictionary keySelector will always exist (otherwise no group would have been materialized), so this is safe. It does look a little odd, but it's the easiest way I can think of to get the key out.

Or as a general purpose function:

async Task<IDictionary<TKey, IList<T>>> ToDictionaryOfLists<T, TKey>(
    IObservable<T> source,
    Func<T, TKey> keySelector)
{        
    return await source.GroupBy(keySelector)
                       .SelectMany(x => x.ToList())
                       .ToDictionary(x => keySelector(x[0]));           
}

Which assuming a class:

public class Site
{
    public int SiteId { get; set; }
}

You could use like:

var result = ToDictionaryOfLists(GetSource(...), x=> x.SiteId);

Marcin Gajda · Answer 2 · 2019-12-20T15:07:29.640

EDIT: I just learned that Task returning Select(...) + Merge() can be replaced by SelectMany(...) so better approach would be:

IDictionary<int, IList<Site>> result = await GetSource()
.GroupBy(o => o.SiteId)
.SelectMany(async group => (group.Key, List: await group.ToList()))
.ToDictionary(group => group.Key, group => group.List);

Original:

I just started learning RX but maybe something like this is good enough:

IDictionary<int, IList<Site>> result = await GetSource()
.GroupBy(o => o.SiteId)
.Select(async group => (group.Key, List: await group.ToList()))
.Merge()
.ToDictionary(group => group.Key, group => group.List);

Assuming Site implementation like:

public class Site
{
   public int SiteId { get; set; }
   //rest of class
}

Theodor Zoulias · Answer 3 · 2019-12-19T20:22:08.743

It seems that you don't actually need a Dictionary but a Lookup<TKey,TElement>.

Represents a collection of keys each mapped to one or more values.

Creating a Lookup is trivial, by using the built-in RX operator ToLookup. Converting a Lookup to a Dictionary of lists is also trivial, by using the standard LINQ operator ToDictionary. Below are both of these operators combined in a single extension method:

public static async Task<Dictionary<TKey, List<TSource>>> ToDictionaryOfLists<TSource, TKey>(
    IObservable<TSource> source, Func<TSource, TKey> keySelector)
{
    var lookup = await source.ToLookup(keySelector);
    return lookup.ToDictionary(g => g.Key, g => g.ToList());
}

How to await on IObservable of IObservables?

3 Answers3