Using Rx to Pair Up Work with Workers

Question

Being fairly new to Rx, I'm trying to achieve something that seems conceptually simple, but I've been struggling for a day trying to figure out how to achieve it in Rx.

I have 2 observables, one is an IObservable<Worker> and the other is IObservable<Work>. The idea is sort of a work balancer setup, where work is submitted through one observable, and paired up with a worker that is submitted on the other observable.

The simple case is easy enough to achieve using Observable.Zip. But there is one additional requirement that throws a wrench in the works.

Each worker is only able to accept work for a limited time (there is a window that closes and the worker is no longer eligible to accept work and should not be paired with a work item).

Here is a marble diagram of what I'm trying to achieve:

-----A-----B-----C------D----->    (WORKERS IObservable<IObservable<Worker>>)
     |     |     |      |
     |     |     |      |
     -----------]|      ----]
           |     |
           |     |----------]
           |
           |-----]


--------1----------2--3------>      (WORK IObservable<Work>)


--------A1---------C2---D3-->       (WORKER/WORK PAIRS IObservable<PairedWork>)

So to recap, the requirements:

When work comes in it should be assigned to the first worker who's window is still active at the time the work comes in.(In the diagram we have the A1 pair but no B2, since worker B is expired by the time work item 2 comes in and C2 becomes the correct pairing)
When a worker is paired up with a work item, it should not be eligible to be paired with subsequent work items (in a sense, its window should be force closed whenever it is used but this is the part I'm most struggling with how to achieve).
If work comes in and there is no available worker with an open window, it should wait until the next worker is emitted and pair with it (see D3 in the diagram).

I'm pretty sure this is somehow achievable with Join or GroupJoin, but I'm struggling to figure out how to get the exact semantics I'm looking for. Any help would be appreciated.

You have this tagged as RxJava, but your conventions look reminiscent of C#. Is this for Java or .NET? — Whymarrh, Dec 18 '15 at 15:18
The particular examples are written in C#, but we are using both languages. I'm more interested in how to solve this issue conceptually using generic RX constructs and not necessarily worried about language-specific syntax. I figured both rx-java and .net communities could contribute. — Daniel Tabuenca, Dec 18 '15 at 15:24
For the timeout of the workers, you might be able to use the [timeout](http://reactivex.io/documentation/operators/timeout.html) operator, since that fits semantically. As for the pairing of the work, I might suggest not trying to shoehorn that into a combination of Rx operators since it seems like that is quite logic heavy. Though if the exercise is to use the operators, [there are other combining operators](http://reactivex.io/documentation/operators.html#combining) that might work better than `Zip`/`GroupJoin` (e.g. `withLatestFrom`). — Whymarrh, Dec 18 '15 at 17:24
For removing a busy worker from the running, its observable could complete when it gets assigned work (if you wanted to make it a custom observable) or you could have a middleman (of sorts) that pairs while keeping track of who is busy (as opposed to using the combining operators). — Whymarrh, Dec 23 '15 at 00:18

score 1 · Answer 1 · answered Dec 19 '15 at 15:33

1

For this task, I don't trust join enough because of the non-deterministic windowing and cross-evaluations. Instead, I'd write a custom zip operator for this where the values of the first sequence is tagged with a time limit value and the zip algorithm simply throws away old values of the first source.

Here is an example class ZipUntil and runnable program in C#.

It is lock-free and uses concepts from Advanced RxJava, although I don't understand how synchronous cancellation works in Rx.NET (because the same pattern doesn't work in RxJava).

answered Dec 19 '15 at 15:33

akarnokd

69,132
14
157
192

I really like this idea, although unless I'm going to use this same pattern a lot, I would just implement a simpler version that doesn't have to deal with concurrencty issues (I can run a background thread and observe both sequences on the same thread) – Daniel Tabuenca Dec 20 '15 at 01:10
Also this gives me the idea of a custom zip operator that takes an expression to evaluate at zip time to discard elements not matching an expression. Something like `Zip(IObs, IObs, Func, Func, Func )` – Daniel Tabuenca Dec 20 '15 at 01:11
I like having greater control over my operators and yes, this allows all kinds of extensions once the structure is understood. – akarnokd Dec 22 '15 at 10:05

score 0 · Answer 2 · answered Dec 18 '15 at 21:01

I doubt this is optimal, but I think it would work.

public static IObservable<Tuple<Worker, Work>> PairWorkers(IObservable<IObservable<Worker>> workerObservable, IObservable<Work> workObservable)
{
    var joined = workerObservable.Join(workObservable,
        worker => worker,
        work => Observable<Work>.Never(),
        (worker, work) => Tuple.Create(worker.FirstAsync(), work)
    );
    var distinct = joined.Distinct(t => t.Item2);
    return distinct;
}

Using Rx to Pair Up Work with Workers

2 Answers2