0

I'd like to measure how many events arrive within allowed lateness grouped by particular feature of the event. We assume particular type of events have way more late arrivals and would like to verify this.

The place to make the measurement I thought of is our custom trigger within onElement method as this is the place where we know whether event is late of not. Yet in case of SlidingEventTimeWindows that means that a single element can be counted multiple timess if it's late by more than a slide.

Any suggestions?

sumek
  • 26,495
  • 13
  • 56
  • 75

1 Answers1

0

You might do this separately from the windowing. You could set the allowed lateness to zero, and divert all late events to a side output. You can then key that stream of late events by the feature(s) of interest, and use a RichFlatMapFunction or KeyedProcessFunction to count the events, which can then be reported as a custom metric, or sent to a sink.

David Anderson
  • 39,434
  • 4
  • 33
  • 60
  • This solution means I wouldn't get the late firings, no? I would still like the late elements to be processed. – sumek Mar 18 '21 at 14:26
  • In that case you could do this completely independently of the windowing. E.g., put a process function in front of the window, and use it to detect and count late events. Results go into a metric, or can be sent to a side output. – David Anderson Mar 18 '21 at 15:21