I want to implement Event Alert using Flink CEP.
My use case: I want to apply time Window for one hour and within that time if device usage is greater than some threshold(100) then I want to alert the end user and fire that alert only once in particular time window.
Problems:
- By time window, Pattern stream gets elements after time window completion. So it doesn't work in real time scenario. Is there any solution?
- Can I get element in Pattern stream before time window?
- How can I alert only once if matching pattern found for the end user?
SourceStream:
KeyedStream<UsageStatistics, Tuple> keyedStreamByMac = kafkaSourceStream.keyBy(UsageStatisticsKey.clusterId.name(), UsageStatisticsKey.deviceMac.name());
SingleOutputStreamOperator<MacTotalUsage> macStream = keyedStreamByMac.timeWindow(Time.hours(1)).reduce(new ReduceFunction<UsageStatistics>() {
@Override
public UsageStatistics reduce(UsageStatistics value1, UsageStatistics value2) throws Exception {
UsageStatistics statistics = new UsageStatistics();
statistics.setDeviceMac(value1.getDeviceMac());
statistics.setDownloadBytes(value1.getDownloadBytes() + value2.getDownloadBytes());
statistics.setUploadBytes(value1.getUploadBytes() + value2.getUploadBytes());
statistics.setClusterId(value1.getClusterId());
return statistics;
}
}).map(new MapFunction<UsageStatistics, MacTotalUsage>() {
@Override
public MacTotalUsage map(UsageStatistics value) {
MacTotalUsage macTotalUsage = new MacTotalUsage();
macTotalUsage.setClusterId(value.getClusterId());
macTotalUsage.setDeviceMac(value.getDeviceMac());
macTotalUsage.setTotalDownloadBytes(value.getDownloadBytes());
macTotalUsage.setTotalUploadBytes(value.getUploadBytes());
return macTotalUsage;
}
});
Pattern:
Pattern<MacTotalUsage, MacTotalUsage> maxUsagePattern = Pattern.<MacTotalUsage>begin("first").where(new SimpleCondition<MacTotalUsage>() {
@Override
public boolean filter(MacTotalUsage event) throws Exception {
return (event.getTotalDownloadBytes() >= 100);
}
});
PatternStream:
DataStreamOperator<MaxUsageAlert> maxUsageAlert = CEP.pattern(macStream, maxUsagePattern).select(new PatternSelectFunction<MacTotalUsage, MaxUsageAlert>() {
@Override
public MaxUsageAlert select(Map<String, List<MacTotalUsage>> pattern) throws Exception {
MacTotalUsage event = pattern.get("first").get(0);
return new MaxUsageAlert(event);
}
});
It's really helpful if anyone gives the solution.