How do I run multiple queries in the same StreamInsight process?

Question

I've been working with StreamInsight for a few weeks now and generally have it doing what I want it to do with one exception. I can't get multiple queries to run in the same process. E.g., given that eventStream is IQStreamable<eventClass>:

var query1 = from e in eventStream
                    where e.EventCode == "33"
                    select e;    
var query2 = from e in eventStream
                        where e.EventCode == "1"
                        select e;
var query1Observable = from e in query1.ToPointObservable()
                        where e.EventKind == EventKind.Insert
                        select e.Payload.EventCode + ": " + e.Payload.EventDescription;
var query2Observable = from e in query2.ToPointObservable()
                        where e.EventKind == EventKind.Insert
                        select e.Payload.EventCode + ": " + e.Payload.EventDescription;
var query1Observer = myApp.DefineObserver(() => Observer.Create<string>(Console.WriteLine));
var query2Observer = myApp.DefineObserver(() => Observer.Create<string>(Console.WriteLine));
var binding = query1Observable.Bind(query1Observer).With(query2Observable.Bind(query2Observer));

using (binding.Run())
{
    System.Console.WriteLine("Press enter to stop at any time.");
    System.Console.ReadLine();
}

When this runs only about half of query1's events are sent to query1Observer and only about half of query2's events are sent to query2Observer. The others just disappear. If I just make my binding

query1Observable.Bind(query1Observer)

then all of the query's results are notified to the observer - but only for query1 of course. How do I run 2 queries, notifying different observers in the same process?

score 0 · Answer 1 · answered Apr 10 '15 at 14:02

0

Have you run a trace to see what's going on? There's nothing that immediately strikes me as incorrect about your queries and it should work without issue. However, keep in mind that Console.WriteLine is static, single threaded and not the most accurate way to really "see" all events that are happening. It may well be that the events are sent but are being dropped by the observer because of locks.

answered Apr 10 '15 at 14:02

DevBiker

451
2
4

Thanks, could you clarify what you mean by Trace? Using the StreamInsight Event Flow debugger? I wondered about whether threading was an issue but if I use `query1Observable.Bind(query1Observer).With(query2Observable.Bind(query1Observer))` i.e. bind both queries to the same observer, I get the same result - not all events are written to the console. In production I want to bind multiple observers to the same query but I can't get that to work consistently, hence reducing to a simple Console.WriteLine – Graeme Wilson Apr 10 '15 at 15:01

score 0 · Answer 2 · answered Apr 14 '15 at 11:15

@DevBiker, thanks for the heads up on trace.cmd - it let me see the problem, which was that each query was only receiving half of the events in the stream. This was a problem in my Observable<T> source which is receiving the events from an Azure Topic. What I hadn't realised is that the .With extension creates two instances of the source Observable<T>. Each source was getting the subscription Id from the app.config so each Observable had an instance of a SubscriptionClient with the same subscription Id. This meant that each instance of the SubscriptionClient was receiving half of the events.

How do I run multiple queries in the same StreamInsight process?

2 Answers2