2

I want to implement Event Alert using Flink CEP.

My use case: I want to apply time Window for one hour and within that time if device usage is greater than some threshold(100) then I want to alert the end user and fire that alert only once in particular time window.

Problems:

  1. By time window, Pattern stream gets elements after time window completion. So it doesn't work in real time scenario. Is there any solution?
  2. Can I get element in Pattern stream before time window?
  3. How can I alert only once if matching pattern found for the end user?

SourceStream:

KeyedStream<UsageStatistics, Tuple> keyedStreamByMac = kafkaSourceStream.keyBy(UsageStatisticsKey.clusterId.name(), UsageStatisticsKey.deviceMac.name());

SingleOutputStreamOperator<MacTotalUsage> macStream = keyedStreamByMac.timeWindow(Time.hours(1)).reduce(new ReduceFunction<UsageStatistics>() {
            @Override
            public UsageStatistics reduce(UsageStatistics value1, UsageStatistics value2) throws Exception {
                UsageStatistics statistics = new UsageStatistics();
                statistics.setDeviceMac(value1.getDeviceMac());
                statistics.setDownloadBytes(value1.getDownloadBytes() + value2.getDownloadBytes());
                statistics.setUploadBytes(value1.getUploadBytes() + value2.getUploadBytes());
                statistics.setClusterId(value1.getClusterId());
                return statistics;
            }
        }).map(new MapFunction<UsageStatistics, MacTotalUsage>() {

            @Override
            public MacTotalUsage map(UsageStatistics value) {
                MacTotalUsage macTotalUsage = new MacTotalUsage();
                macTotalUsage.setClusterId(value.getClusterId());
                macTotalUsage.setDeviceMac(value.getDeviceMac());
                macTotalUsage.setTotalDownloadBytes(value.getDownloadBytes());
                macTotalUsage.setTotalUploadBytes(value.getUploadBytes());
                return macTotalUsage;
            }
        });

Pattern:

Pattern<MacTotalUsage, MacTotalUsage> maxUsagePattern = Pattern.<MacTotalUsage>begin("first").where(new SimpleCondition<MacTotalUsage>() {
            @Override
            public boolean filter(MacTotalUsage event) throws Exception {
                return (event.getTotalDownloadBytes() >= 100);
            }
        });

PatternStream:

DataStreamOperator<MaxUsageAlert> maxUsageAlert = CEP.pattern(macStream, maxUsagePattern).select(new PatternSelectFunction<MacTotalUsage, MaxUsageAlert>() {
    @Override
    public MaxUsageAlert select(Map<String, List<MacTotalUsage>> pattern) throws Exception {
        MacTotalUsage event = pattern.get("first").get(0);
        return new MaxUsageAlert(event);
    }
});

It's really helpful if anyone gives the solution.

NIrav Modi
  • 6,038
  • 8
  • 32
  • 47
  • Anyway I think in the same way as Table/SQL API is built on top of the Streaming API, you can achieve this with a process function. Operating on event timestamps you can achieve the "windowing" you need. – Alex Apr 16 '18 at 14:47
  • Hey Alex, Is it proper way if I collect element in processElement method? – NIrav Modi Apr 17 '18 at 15:21

0 Answers0