StreamInsight complex query with filter on HoppingWindow

Question

I'm trying out StreamInsight and I came across a problem with a query I need.

I'm trying to throw a warning if there are several changes in my measured values (of up to 20% change) in the last 30 minutes.

This is the query I came up with for now but it isn't working and it's not even correct I think.

Apparently I can't filter on a window...?

var deviationQuery = from s in wcfStream
                     group s by s.SensorId into grouped
                     from window in grouped.HoppingWindow(TimeSpan.FromMinutes(30),TimeSpan.FromMinutes(1))
                     where window.StdDev(e => e.Value) > measurableValue * 1.2
                     select new OutputEvent
                     {
                         Error = "Deviation"
                     };

Thanks in advance!

Well, the HoppingWindow should be on a CepStream (http://msdn.microsoft.com/en-us/library/microsoft.complexeventprocessing.linq.cepstream.hoppingwindow.aspx). But if I do wcfStream.HoppingWindow(...), I can't do any grouping afterwards, so I have no clue how I have to combine a grouping and a window. And weirdly enough, the code builds perfectly fine with the hoppingwindow on the grouping ... — hhoud, Oct 10 '12 at 13:18
What is it doing incorrectly? I deleted my previous comment because it was incorrect. — Risky Martin, Oct 10 '12 at 13:46
Maybe you shouldn't look at my query and show me how you would take on the problem. THE PROBLEM: In a window of 30 minutes, hopping every minute, check how many measured values are wrong (lets say bigger than 50) and send a warning if there are more than five. — hhoud, Oct 10 '12 at 14:39

Risky Martin · Answer 1 · 2012-10-10T15:34:23.803

1

If I understand correctly, this is what you want to do:

Group the stream by SensorId.
Divide each group into 30-minute windows.
Write an error message for the windows that have too many incorrect values.

This should do it, hopefully.

var deviationQuery = from s in wcfStream
                     group s by s.SensorId into grouped
                     from window in grouped.HoppingWindow(TimeSpan.FromMinutes(30),TimeSpan.FromMinutes(1))
                     where window.Count(event => event.Value > maxValue) > maxIncorrectValues                        
                     select new OutputEvent
                     {
                         Error = "Deviation"
                     };

edited Oct 10 '12 at 15:34

answered Oct 10 '12 at 14:57

Risky Martin

2,491
2
15
16

This is almost what I need, the only thing that has to change is that I don't need the StdDev to be compared but the CURRENT value of the measurement coming in. But I don't know how I can get to the current value from that point. So instead of `w => w.StdDev(e => e.Value)` it should be `e => e.Value`but you can't do that... – hhoud Oct 10 '12 at 15:10
For some reason I get following error now: `Could not find an implementation of the query pattern for source type 'Microsoft.ComplexEventProcessing.Linq.IQPartitionedStreamable'. 'Select' not found. ` – hhoud Oct 11 '12 at 06:53

score 1 · Accepted Answer · answered Oct 11 '12 at 07:40

I found a working query for my problem. At first I thought it didn't work but I misinterpreted the results. It may not be the shortest and best query, so if you have a better answer, please tell me!

var deviationQuery = from s in wcfStream
                     where s.Value > measurableValue * (1 + deviationThreshold) || s.Value < measurableValue * (1 - deviationThreshold)
                     group s by s.SensorId into grouped
                     from window in grouped.HoppingWindow(TimeSpan.FromSeconds(180), TimeSpan.FromSeconds(120))
                     select window.Count();
var deviation = from c in deviationQuery
                where c > maxIncorrectValues
                select new OutputEvent
                    {
                        M = new Measurement() { SensorId = "354354", Value = 53, Time = DateTime.Now },
                        Deflection = c,
                        Error = "Deviation"
                    };

This solution looks great! It is most likely more efficient to filter out values before grouping them like you did. If you want to trade a small amount of readability for performance you can change `select window.Count()` to `select window.Skip(maxIncorrectValues).Any()`, which will stop iterating through a window's events once it goes over the limit. Note that it will return a bool that will be true if the window is a deviation. The other thing you could do is combine the queries to save a couple lines of code, but I'm guessing that's what was causing the error. — Risky Martin, Oct 11 '12 at 13:33

StreamInsight complex query with filter on HoppingWindow

2 Answers2