How to control flink sending output to sideoutput in keyedbroadcastprocessfunction

Question

I am trying to validate a data stream against a set of rules to detect patterns in flink by validating the data stream against a broadcast stream with set of rules i using for loop to collect all the patterns in map and iterating through it in processElement fn to find a pattern sample code is as below

MapState Descriptor and Side output stream as below

public static final MapStateDescriptor<String, String> ruleSetDescriptor =
        new MapStateDescriptor<String, String>("RuleSet", BasicTypeInfo.STRING_TYPE_INFO,
                BasicTypeInfo.STRING_TYPE_INFO);

public final static OutputTag<Tuple2<String, String>> unMatchedSideOutput =
        new OutputTag<Tuple2<String, String>>(
                "unmatched-side-output") {
        };

Process Function and Broadcast Function as below:

@Override
    public void processElement(Tuple2<String, String> inputValue, ReadOnlyContext ctx,
                               Collector<Tuple2<String,
                                       String>> out) throws Exception {


        for (Map.Entry<String, String> ruleSet:
                ctx.getBroadcastState(broadcast.patternRuleDescriptor).immutableEntries()) {

            String ruleName = ruleSet.getKey();


//If the rule in ruleset is matched then send output to main stream and break the program
            if (this.rule) {
                out.collect(new Tuple2<>(inputValue.f0, inputValue.f1));
                break;
            }
        }

        // Writing output to sideout if no rule is matched 
        ctx.output(Output.unMatchedSideOutput, new Tuple2<>("No Rule Detected", inputValue.f1));
    }


    @Override
    public void processBroadcastElement(Tuple2<String, String> ruleSetConditions, Context ctx, Collector<Tuple2<String,String>> out) throws Exception {

        ctx.getBroadcastState(broadcast.ruleSetDescriptor).put(ruleSetConditions.f0,
                ruleSetConditions.f1);

    }

I am able to detect the pattern but i am getting sideoutput also since i am trying to iterate over the rules one by one if my matched rule is present in last, the program is sending output to sideoutput since the initial set of rules won't match. I want to print sideoutput only once if none of the rules are satisfied, i am new to flink please help how can i achieve it.

Can you clarify what is wrong with what you are currently doing? — David Anderson, Jun 01 '20 at 13:12
@DavidAnderson the problem here let say i got a data stream which is having a valid data let say i have 5 rules on which the input stream looped to validated and let say the rule for which the stream is going to satisfy the condition is at 5th place by the time i reach 5th place the loop will print its invalid signal as 4 times and push it to DeadLetterQueue(DLQ) but i want to do this only after checking all rules and if not found valid rule — YRK, Jun 01 '20 at 13:52
@DavidAnderson Any suggestion here to achieve this solution for this issue? — YRK, Jun 02 '20 at 13:47

score 0 · Answer 1 · answered Jun 05 '20 at 09:53

It looks to me like you want to do something more like this:

@Override
public void processElement(Tuple2<String, String> inputValue, ReadOnlyContext ctx, Collector<Tuple2<String, String>> out) throws Exception {

    transient boolean matched = false;

    for (Map.Entry<String, String> ruleSet:
      ctx.getBroadcastState(broadcast.patternRuleDescriptor).immutableEntries()) {

        String ruleName = ruleSet.getKey();

        if (this.rule) {
            matched = true;
            out.collect(new Tuple2<>(inputValue.f0, inputValue.f1));
            break;
        }
    }

    // Writing output to sideout if no rule was matched
    if (!matched) {
        ctx.output(Output.unMatchedSideOutput, new Tuple2<>("No Rule Detected", inputValue.f1));
    }
}

How to control flink sending output to sideoutput in keyedbroadcastprocessfunction

1 Answers1