0

I am working with apache flink v1.4 and we want to fire our window using event time.

I have made sure that:

  1. Time Characteristic have been set env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime); is present

  2. I am assigning the watermarks using following code

deviceVnetSource.assignTimestampsAndWatermarks(WatermarkStrategy.<DataEngineMessage<String, DeviceVnet>>forMonotonousTimestamps().withTimestampAssigner((event,timestamp)-> event.getValue().getTsSec() * 1000))
  1. Time Value is in milliseconds

  2. In the Sink Function I see following output when events are passed (my watermark time is increasing)

  3. TestSinkFunction - watermark time - -9223372036854775808
    
    TestSinkFunction - watermark time - 1674113291999 
    
    TestSinkFunction - watermark time - 1674113292999 
    
    TestSinkFunction - watermark time - 1674113301999 
    
    TestSinkFunction - watermark time - 1674113319999 
    
    

But My Window function is not triggering which is set to 2 seconds.

Can someone please tell me where I am going wrong?

This is how entire pipeline looks like -

public class FlinkJob { 
  @Autowired
   StreamExecutionEnvironment env;

 private void runJob() throws Exception {
   env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
  
 var deviceVnetSource = 
  streamClient.streamFromTopic(Constants.DEVICEVNET_SOURCE_TOPIC, 
    "DeviceVnet Source");
   
 var unifiedDistanceStream =      
  deviceVnetSource.assignTimestampsAndWatermarks(WatermarkStrategy. 
<DataEngineMessage<String, DeviceVnet>>forMonotonousTimestamps()
            .withTimestampAssigner((event,timestamp)-> 
     event.getValue().getTsSec() * 1000))
            //.keyBy(key -> key.getKey())
            //.window(TumblingEventTimeWindows.of(Time.seconds(1))) This window is not triggering at all
            //.process(new UnifiedDistanceOperatorTest())
            .addSink(new TestSinkFunction());
}
}
Anup
  • 73
  • 2
  • 10
  • Can You provide some code for Your Pipeline ? – Dominik Wosiński Jan 19 '23 at 10:18
  • @dominik-wosiński i have added the pipeline code. – Anup Jan 19 '23 at 15:03
  • What happens if You uncomment `keyBy` and try to check if then the watermark is increasing? – Dominik Wosiński Jan 19 '23 at 19:18
  • @DominikWosiński it worked once i uncommented key by as well as window part. Thing which helped was adding this line - `env.setParallism(1)` but same code base is not working if i introduce a mapping operation on `deviceVnetSource` stream and get a mapped stream and then apply watermarking. – Anup Jan 20 '23 at 15:01
  • `env.setParallelism(1)` suggests the anser here, some of the partitions do not have data flowing through it and they are causing the watermark to stay without change. – Dominik Wosiński Jan 20 '23 at 15:27

0 Answers0