0

I am working on use case wherein I have two unbounded streams and want to do left join on these streams. Using a fixed-size window of 5 minutes with no allowed lateness.For join I am using java extension join library. But After Join it's not emitting results. Code for same:

PCollection<KV<String, KV<GenericRecord, GenericRecord>>> joinedDatasets = Join.leftOuterJoin(aById, bById, GenericRecord);
        PCollection<GenericRecord> result = joinedDatasets.apply(ParDo.of(new DoFn<KV<String, KV<GenericRecord, GenericRecord>>, GenericRecord>() {
            @ProcessElement
            public void processElement(@Element KV<String, KV<GenericRecord, GenericRecord>> element, OutputReceiver<GenericRecord> out) {
                LogHelper.info(element.getKey());
                //some processing logic
            }
        }));

I have tried cogrpbykey but behaviour is same.Running job on directrunner

Vadim Kotov
  • 8,084
  • 8
  • 48
  • 62
deep
  • 31
  • 5
  • the pipeline seems to be right. I would recommend checking the element timestamp to make sure they fall in the same window and the pcollection are windowed in fixed window. – Ankur Apr 16 '20 at 00:14
  • 1
    Thanks @Ankur , adding the processing time trigger to the fixed window solved it. – deep Apr 17 '20 at 08:29

0 Answers0