2

I want to group data by time with Siddhi. I have a test based on https://docs.wso2.com/display/SP400/Incremental+Analysis.

@App:name("DataAggregation")
@App:description("Incremental aggregation")
define stream InputStream (sensorid long, value double, timestamp long);
@store( type="rdbms", jdbc.url="jdbc:mysql://localhost:3306/test", username="root", password="xxx", jdbc.driver.name="com.mysql.jdbc.Driver")
define aggregation DataAggregation from InputStream select sensorid, sum(value) as total group by sensorid aggregate by timestamp every hour ... month;

I send multiple events:

sensorid:1, value: 1.0, timestamp:2018-09-10 01:00
sensorid:1, value: 2.0, timestamp:2018-09-10 02:00
...
sensorid:1, value: 23.0, timestamp:2018-09-10 23:00
sensorid:1, value: 24.0, timestamp:2018-09-11 00:00
sensorid:1, value: 25.0, timestamp:2018-09-11 01:00
...

Siddhi saves hourly data and it calculates daily data. If then I send some events of August month:

sensorid:1, value: 10.0, timestamp:2018-08-10 01:00
sensorid:1, value: 20.0, timestamp:2018-08-10 02:00
...
sensorid:1, value: 230.0, timestamp:2018-08-10 23:00
sensorid:1, value: 240.0, timestamp:2018-08-11 00:00
sensorid:1, value: 250.0, timestamp:2018-08-11 01:00

Why are August events ignored and Siddhi does not store them?

Thanks

Javier
  • 21
  • 1

1 Answers1

1

In Siddhi 4.x.x - 4.1.x (Used in Stream Processor 4.0.0 - 4.2.0), incremental aggregation out of order event handling is handled using buffers for the lowest granularity. In the above aggregation, it's for the hour. However, in your annotation definition @BufferSize annotation is not defined, thus all out of order events are dropped. In your case, the out of order events are late by a month, these cannot be handled by the use of buffers.

Please try the same annotation in WSO2 SP 4.3.0-rc1, which uses the latest Siddhi 4.2.x, in which out of order events are calculated internally and w/o the use of buffers. Thus the above scenario will give accurate aggregations. Please note the WSO2 SP 4.3.0 GA will only be released on Sept 15 and the above pack is only a release candidate but can be used for trial.

Niveathika
  • 1,319
  • 2
  • 8
  • 17