6

For incoming record I need to validate the value and based on result object I need to forward error to different topics and if successfully validated then forward the same using context.forward(). It can be done using DSL as provided in this link

using kafka-streams to conditionally sort a json input stream

I am not finding a clear way of doing this in processorAPI.

    ValidateProcessor.java

    @Override
    public void process(String key, String value) {
        Object result = //validation logic
        if(result.isSuccessful()) {
            context().forward(key, value);
         }else {
            context.forward("error",Object)
        }

}

Now the caller again need to check and based on key need to differentiate the sink topic. I am using processorAPI because I need use headers.

Edit :

branch(new predicate{
 business logic 
 if(condition)
   return true
 else
   return false;

When the condition is false how to push to different stream. Currently creating another predicate which collects all other records which doesn't satisfy the above predicate in chain. Is there a way to do in same predicate?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
vineet gandhi
  • 77
  • 1
  • 7

1 Answers1

9

When you specify your Topology, you assign names to all node and connect them:

Topology topology = new Topology();
topology.addSource("source", ...);
topology.addProcessor("X", ..., "source"); // connect source->X
topology.addSink("Y", ..., "X"); // connect X->Y
topology.addSink("Z", ..., "X"); // connect X->Z

If a processor "X" is connected to downstream processors "Y" and "Z", you can use the node name to send a record to either "Y" or "Z". If you don't specify a name, the record is send to all downstream ("child") processors.

// this is `process()` of "X"
public void process(String key, String value) {
    context.forward(newKey, newValue); // send to both Y and Z
    context.forward(newKey, newValue, To.child("Y")); // send it only to Y
    context.forward(newKey, newValue, To.child("Z")); // send it only to Z
}
Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
  • I used predicate in streamsBuilder and branch(). If predicate is evaluated to true then the records are send to one branched streams. But in the same computation of predicate I wanted to some business object if the predicate is evaluated false but i am not sure this will work. – vineet gandhi Feb 24 '19 at 13:26
  • This question was about Processor API. `StreamsBuilder` and `branch()` are part of the DSL. You should post a new question. – Matthias J. Sax Feb 24 '19 at 20:45
  • If i don't forward the context, is the record sent to all downstream ("child") processors or none? – Arbaz Sheikh May 18 '21 at 10:08
  • If you don't call `forward()` no records will be send downstream. For example, if you implement a filter, and want to drop a record, you would skip calling `forward()`. – Matthias J. Sax May 18 '21 at 15:18