How to match events in two causal streams to detect excessive lagging

Question

I have two hot observables, which are respectively a stream Q of requests to a network server, and a stream R of replies from the server. The replies are always delivered in the order of requests, and every request is going to receive exactly one reply eventually. Thus the first event in R, R₁, is the reply to the first event in Q, Q₁, and so on. I need to detect when a reply Rn takes longer than a defined timeout and signal this timeout condition.

  Q --1---2---------3------->  // Requests Q1, Q2...
  R ----1------------------->  // Replies
Out ------------------O-|>     // Oops: Reply R2 to Q2 did not arrive within time τ.
          |<----τ---->|

Events Qn and Rn do not contain any identifying information (think of plain colorless round marbles), and the indices in the diagram are just sequential numbers introduced for explanation.

I seem unable to solve this riddle. I tried the approach below, but it appears I am matching the latest request Q_i to the latest response R_j. In the sample Q contains 5 requests, spaced 500ms apart, and replies in R come 750ms apart, starting at 200ms, but only 4 of them (the 5th is delayed indefinitely). The code does not detect that, since that last reply R₄ comes within the set timeout of 1000ms after the latest request Q₅ (in 200ms, actually).

var Q = Observable.Interval(TimeSpan.FromMilliseconds(500)).Select(_ => Unit.Default)
                   .Take(5).Concat(Observable.Never<Unit>());
var R = Observable.Interval(TimeSpan.FromMilliseconds(750)).Select(_ => Unit.Default)
                   .Delay(TimeSpan.FromMilliseconds(200))
                   .Take(4).Concat(Observable.Never<Unit>());

var dq = Q.Select(v => Observable.Return(v).Delay(TimeSpan.FromMilliseconds(1000)));
var dr = Observable.Zip(Q, R, (_1,_2) => Observable.Never<Unit>());
Observable.Merge(dq, dr).Dump().Switch().Dump();

score 1 · Accepted Answer · edited Jul 28 '16 at 04:21

1

I believe that you want to be notified that request 4 has timed out (due at 3s, but arrives at 3.2s) and also request 5 as it never arrives

void Main()
{
    var scheduler = new TestScheduler();
    var requests = scheduler.CreateHotObservable<int>(
        ReactiveTest.OnNext(0500.Ms(), 1),
        ReactiveTest.OnNext(1000.Ms(), 2),
        ReactiveTest.OnNext(1500.Ms(), 3),
        ReactiveTest.OnNext(2000.Ms(), 4),
        ReactiveTest.OnNext(2500.Ms(), 5));
    var responses = scheduler.CreateHotObservable<Unit>(
        ReactiveTest.OnNext(0950.Ms(), Unit.Default),
        ReactiveTest.OnNext(1700.Ms(), Unit.Default),
        ReactiveTest.OnNext(2450.Ms(), Unit.Default),
        ReactiveTest.OnNext(3200.Ms(), Unit.Default));

    var expected = scheduler.CreateHotObservable<int>(
            ReactiveTest.OnNext(3000.Ms(), 4),
            ReactiveTest.OnNext(3500.Ms(), 5)
            );

    var observer = scheduler.CreateObserver<int>();

    var query = responses
        .Select((val, idx)=>idx)
        .Publish(responseIdxs =>
    {
        return requests.SelectMany((q, qIdx) =>
                Observable.Timer(TimeSpan.FromSeconds(1), scheduler)
                    .TakeUntil(responseIdxs.Where(rIdx => qIdx == rIdx))
                    .Select(_ => q));
    });

    query.Subscribe(observer);
    scheduler.Start();

    //This test passes
    ReactiveAssert.AreElementsEqual(
        expected.Messages,
        observer.Messages);

}

// Define other methods and classes here
public static class TickExtensions
{
    public static long Ms(this int ms)
    {
        return TimeSpan.FromMilliseconds(ms).Ticks;
    }
}

edited Jul 28 '16 at 04:21

kkm inactive - support strike

5,190
2
32
59

answered Jul 27 '16 at 07:17

Lee Campbell

10,631
1
34
29

He said that there's nothing identifying about qn or rn, so you can't use q == r. You have to match by index (use indexed SelectMany on requests, then use index to skip responses), or cheat and add an identifier somewhere. – Shlomo Jul 27 '16 at 08:33
But he also states that the responses will always be in order, so the conditional check is valid – Lee Campbell Jul 27 '16 at 08:34
... For my sample data. But yes, he could switch out to use the index from `Select` – Lee Campbell Jul 27 '16 at 08:38
If you switch the integers to `Unit.Default`, like the question, the test fails. – Shlomo Jul 27 '16 at 13:41
If you have `{q1@0, q2@500, r1@800, r2@2000}`, and use `Unit.Default`, your code will pass it, because the `Unit` returned `r1@800` will satisfy both `q1` and `q2` requirements, when it should only satisfy `q1`. – Shlomo Jul 27 '16 at 13:43
@Shlomo: It's too easy to number messages in both stream with `.Select((_, n) => n)` – kkm inactive - support strike Jul 27 '16 at 18:45
@LeeCampbell: Thanks for the update, and I too noticed that `responseIdxs` must be published hot. I'm taking a liberty of adding an SO language tag to your answer so that syntax highlighting works. – kkm inactive - support strike Jul 28 '16 at 04:21

How to match events in two causal streams to detect excessive lagging

1 Answers1