0

I have a stream of events and want to count the number of events in a specific period time to find out event loss. My code is something similar to the following:

DataStream<DataEvent> dataStream = ...;

dataStream
.windowAll(SlidingEventTimeWindows.of(Time.seconds(windowSize),Time.seconds(1)))
.process(new MyProcessWindowFunction());

and I defined the MyProcessWindowFunction class as:

public class MyProcessWindowFunction
        extends ProcessAllWindowFunction<DataEvent, String, TimeWindow> {

    @Override
    public void process(ProcessAllWindowFunction<DataEvent, String, TimeWindow>.Context context, Iterable<DataEvent> iterable, Collector<String> collector) throws Exception {
        long count = 0;
        for (DataEvent dataEvent : iterable) {
            count++;
        }
        if ()
        collector.collect("Window: " + context.window() + "count: " + count);
    }
}

My question is how can I use the counted value to compare it and find the event loss. As I understand correctly, this process function will create a stream of strings which is collected by the collector. But, I want I want to do something as soon as I found the event loss at the end of each sliding window.

I appreciate any help. Best regards,

Majid Lotfian Delouee
  • 5
  • 2
  • It's not clear what you intend to compare the count to -- how are you going to measure event loss? Where do you suspect events are being lost? Is there some source of truth somewhere else? – David Anderson Apr 24 '22 at 13:12
  • Hi, I have a sensor which generate a data each second and this sensor has a loss rate that based on it, I refuse to send the data. I wanna apply a window-based analysis on this data stream and count the number of events in a window (e.g., number of events in a 50 seconds window) and stop the sensor from sending data once the number of events in the window do not match the loss rate of the sensor. I know that loss rate is something related to links not the sensor, but I assume it as a metric to choose a sensor. Now, after counting the events, I want to trigger an alarm to stop the sensor. – Majid Lotfian Delouee Apr 25 '22 at 07:29

1 Answers1

0

It sounds like you want to do something along these lines:

datastream
  .map(new Tuple2<>(event.sensorId, 1))
  .keyBy(t -> t.f0)
  .window(SlidingEventTimeWindows.of(Time.seconds(windowSize),Time.seconds(1)))
  .reduce(t1, t2 -> new Tuple2<>(t1.f0, t1.f1 + t2.f1)) 
  .filter(t -> new SensorShouldBeStopped(t))
  .addSink(...);
David Anderson
  • 39,434
  • 4
  • 33
  • 60