0

In Scala, I'm using import org.apache.kafka.streams.KafkaStreams and am able to read off of an input stream and easily do some computation and send to an output stream. Is there a way via branch or filter to take the resulting record from the input stream and send to two output streams?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
k1llingm3smalls
  • 37
  • 7
  • Do you wan to split or broadcast? For splitting, using `branch()` is the way to go -- it we put every input record into at most one output stream (if no predicate matches, `branch()` would drop the record). If you want to multi- or broadcast, `branch()` would not work: cf: https://stackoverflow.com/questions/42388525/streaming-messages-to-multiple-topics – Matthias J. Sax Feb 13 '20 at 04:26

1 Answers1

1

branch does exactly what you want. It returns an array of KStream, which you can individually send to() two different topics.

If you want to send the same stream to two topics, use through followed by to

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • How would you use it? Seems like all the examples call for the use of a Predicate of the object in question from the input stream. i.e.: `builder.stream[String, Object](INPUT_TOPIC) .mapValues(v => ValidationFormat.from(v.asInstanceOf[GenericRecord])) .mapValues(someResult => { ..... })` so if the result from the mapValues is a List of two Option[String] elements and if the first element is empty I want to send it to topicA if the second is empty send to topicB – k1llingm3smalls Feb 13 '20 at 00:12
  • 1
    Yes, you use an array of predicate objects. And like I've said before, the schema registry should be responsible for Avro validation. You'd make one predicate for not empty, then the last predicate is just true to catch the rest of the events. You'd use branch after that mapValues, then assign the result to a variable for an array of KStreams objects then you write two separate lines of `to` calls and indexing the array, as shown by the examples – OneCricketeer Feb 13 '20 at 01:56