How to combine two different GroupedStreams in Rx.NET?

Question

This question is similar, but it does not apply to my case, since the user needed the merge observable streams from the same IGroupedObservable, while I want to combine streams from different groups.

I have the following structures and streams:

type A = {
  Id: int
  Value: int
}

type B = {
  Id: int
  Value: int
}

//subjects to test input, just any source of As and Bs
let subjectA: Subject<A> = Subject.broadcast
let subjectB: Subject<B> = Subject.broadcast

//grouped streams
let groupedA: IObservable<<IGroupedObservable<int, A>> = Observable.groupBy (fun a -> a.Id) subjectA
let groupedB: IObservable<<IGroupedObservable<int, B>> = Observable.groupBy (fun b -> b.Id) subjectB

I want to somehow merge the internal observables of A and B when groupedA.Key = groupedB.Key, and get an observable of (A, B) pairs where A.Id = B.Id

The signature I want is something like IObservable<IGroupedObservable<int, A>> -> IObservable<IGroupedObservable<int, B>> -> IObservable<IGroupedObservable<int, (A, B)>> where for all (A, B), A.Id = B.Id

I tried a bunch of combineLatest, groupJoin, filters and maps variations, but with no success.

I'm using F# with Rx.Net and FSharp.Control.Reactive, but if you know the answer in C# (or any language, really) please post it

Do you want a complete cross join? With the only requirement that a.Id == b.Id? — Shlomo, Apr 01 '21 at 23:19
Not sure what "complete cross join" means in this context, but yes, I believe so. — fnzr, Apr 01 '21 at 23:34
I think `Observable.groupBy` returns an `IObservable>`, not just a single `IGroupedObservable<_, _>`, so I assume that's what you're actually trying to combine? — Brian Berns, Apr 02 '21 at 00:34

Theodor Zoulias · Accepted Answer · 2021-04-03T03:08:03.703

Here is a custom operator GroupJoin that you could use. It is based on the Select, Merge, GroupBy and Where operators:

/// <summary>
/// Groups and joins the elements of two observable sequences, based on common keys.
/// </summary>
public static IObservable<(TKey Key, IObservable<TLeft> Left, IObservable<TRight> Right)>
    GroupJoin<TLeft, TRight, TKey>(
    this IObservable<TLeft> left,
    IObservable<TRight> right,
    Func<TLeft, TKey> leftKeySelector,
    Func<TRight, TKey> rightKeySelector,
    IEqualityComparer<TKey> keyComparer = null)
{
    // Arguments validation omitted
    keyComparer ??= EqualityComparer<TKey>.Default;
    return left
        .Select(x => (x, (TRight)default, Type: 1, Key: leftKeySelector(x)))
        .Merge(right.Select(x => ((TLeft)default, x, Type: 2, Key: rightKeySelector(x))))
        .GroupBy(e => e.Key, keyComparer)
        .Select(g => (
            g.Key,
            g.Where(e => e.Type == 1).Select(e => e.Item1),
            g.Where(e => e.Type == 2).Select(e => e.Item2)
        ));
}

Usage example:

var subjectA = new Subject<A>();
var subjectB = new Subject<B>();

IObservable<IGroupedObservable<int, (A, B)>> query = subjectA
    .GroupJoin(subjectB, a => a.Id, b => b.Id)
    .SelectMany(g => g.Left.Zip(g.Right, (a, b) => (g.Key, a, b)))
    .GroupBy(e => e.Key, e => (e.a, e.b));

score 1 · Answer 2 · answered Apr 02 '21 at 14:27

I'm not clear if this is what you want. So it may be helpful to clarify first with runner code. Assuming the following runner code:

var aSubject = new Subject<A>();
var bSubject = new Subject<B>();

var groupedA = aSubject.GroupBy(a => a.Id);
var groupedB = bSubject.GroupBy(b => b.Id);

//Initiate solution

solution.Merge()
    .Subscribe(t => Console.WriteLine($"(Id = {t.a.Id}, AValue = {t.a.Value}, BValue = {t.b.Value}  )"));

aSubject.OnNext(new A() { Id = 1, Value = 1 });
aSubject.OnNext(new A() { Id = 1, Value = 2 });

bSubject.OnNext(new B() { Id = 1, Value = 10 });
bSubject.OnNext(new B() { Id = 1, Value = 20 });
bSubject.OnNext(new B() { Id = 1, Value = 30 });

Do you want to see the following output:

(Id = 1, AValue = 1, BValue = 10)
(Id = 1, AValue = 2, BValue = 10)
(Id = 1, AValue = 1, BValue = 20)
(Id = 1, AValue = 2, BValue = 20)
(Id = 1, AValue = 1, BValue = 30)
(Id = 1, AValue = 2, BValue = 30)

If that's the case, you can get to solution as follows:

var solution = groupedA.Merge()
    .Join(groupedB.Merge(),
        _ => Observable.Never<Unit>(),
        _ => Observable.Never<Unit>(),
        (a, b) => (a, b)
    )
    .Where(t => t.a.Id == t.b.Id)
    .GroupBy(g => g.a.Id);

I'll caution that there are memory/performance impacts here if this is part of a long-running process. This keeps all A and B objects in memory indefinitely, waiting to see if they can be paired off. To shorten the amount of time they're kept in memory, change the Observable.Never() calls to appropriate windows for how long to keep each object in memory.

Yes, that's what I want to achieve. In my case, I want to match the *latest* A with Id = 1 with the *latest* B with Id = 1 for example, while the old values of A and B with Id=1 can be freed from memory. How would I do that? I'm dont quite get what this Observable.Never is doing, I think. — fnzr, Apr 03 '21 at 12:00
@fnzr this is important info that should probably be included in the question. Could you [edit](https://stackoverflow.com/posts/66911017/edit) the question and add this info there? — Theodor Zoulias, Apr 04 '21 at 06:08

Brian Berns · Answer 3 · 2021-04-02T03:11:35.243

As a start, this has the signature you want:

let cartesian left right =
    rxquery {
        for a in left do
        for b in right do
        yield a, b
    }

let mergeGroups left right =
    rxquery {
        for (leftGroup : IGroupedObservable<'key, 'a>) in left do
        for (rightGroup : IGroupedObservable<'key, 'b>) in right do
        if leftGroup.Key = rightGroup.Key then
            let merged = cartesian leftGroup rightGroup
            yield {
                new IGroupedObservable<_, _> with
                    member __.Key = leftGroup.Key
                    member __.Subscribe(observer) = merged.Subscribe(observer)
            }
    }

However, in my testing, the groups are all empty. I don't have enough Rx experience to know why, but perhaps someone else does.

How to combine two different GroupedStreams in Rx.NET?

3 Answers3