0

I am new to Esper and I am trying to filter the events properties from event streams having multiple events coming with high velocity.

I am using Kafka to send row by row of CSV from producer to consumer and at consumer I am converting those rows to HashMap and creating Events at run-time in Esper.

For example I have events listed below which are coming every 1 second

WeatherEvent Stream:

E1 = {hum=51.0, precipi=1, precipm=1, tempi=57.9, icon=clear, pickup_datetime=2016-09-26 02:51:00, tempm=14.4, thunder=0, windchilli=, wgusti=, pressurei=30.18, windchillm=}

E2 = {hum=51.5, precipi=1, precipm=1, tempi=58.9, icon=clear, pickup_datetime=2016-09-26 02:55:00, tempm=14.5, thunder=0, windchilli=, wgusti=, pressurei=31.18, windchillm=}

E3 = {hum=52, precipi=1, precipm=1, tempi=59.9, icon=clear, pickup_datetime=2016-09-26 02:59:00, tempm=14.6, thunder=0, windchilli=, wgusti=, pressurei=32.18, windchillm=}#

Where E1, E2...EN are multiple events in WeatherEvent

In the above events I just want to filter out properties like hum, tempi, tempm and presssurei because they are changing as the time proceeds ( during 4 secs) and dont want to care about the properties which are not changing at all or are changing really slowly.

Using below EPL query I am able to filter out the properties like temp, hum etc

@Name('Out') select * from weatherEvent.win:time(10 sec) match_recognize ( partition by pickup_datetime? measures A.tempm? as a_temp, B.tempm? as b_temp pattern (A B) define B as Math.abs(B.tempm? - A.tempm?) > 0 )

The problem is I can only do it when I specify tempm or hum in the query for pattern matching.

But as the data is coming from CSV and it has high-dimensions or features so I dont know what are the properties of Events before-hand.

I want Esper to automatically detects features/properties (during run-time) which are changing and filter it out, without me specifying properties of events.

Any Ideas how to do it? Is that even possible with Esper? If not, can I do it with other CEP engines like Siddhi, OracleCEP?

tank
  • 465
  • 8
  • 22

1 Answers1

0

You may add a "?" to the event property name to get the value of those properties that are not known at the time the event type is defined. This is called dynamic property see documentation . The type returned is Object so you need to downcast.

user650839
  • 2,594
  • 1
  • 13
  • 9
  • Thanks for the response, but I am already using ? after the event properties name. In the documentation they mentioned that for dynamic property we must use item? in the query. But the problem is I don't know what is item (property) here. I don't know any of the event properties name and type to write like property_name? in the query. Is there any way such that any EPL query can match pattern for every properties in match_recognize like taking difference between all properties in subsequent events in event stream? – tank Aug 09 '18 at 12:55
  • If you don't even know the property name you need to write a plug-in single row function that looks at the Map object for the event and iterates thru the keys. Pass the event itself to that function. The function can return an object representing differences, for example. – user650839 Aug 10 '18 at 05:17
  • Hi, Thanks for the help! Actually instead of returning difference of properties values between the events I want to develop something like which return the properties in the stream of events which are changing quickly for example I want to check if property P1 , P3 & P4 out of P1, P2 , P3...Pn of event A1, A2, A3....An in event stream S is changing (Non-zero difference) for certain time period lets say 5 minutes then I will just extract only those features i.e only P1, P3 & P4. I know it is not possible with single-row function. Can I use aggregation multi-function in ESPER to solve this? – tank Aug 17 '18 at 21:34