Kafka Streams: one record to multiple records

Question

Given: I have two topics in Kafka let's say topic A and topic B. The Kafka Stream reads a record from topic A, processes it and produces multiple records (let's say recordA and recordB) corresponding to the consumed record. Now, the question is how can I achieve this using Kafka Streams.

KStream<String, List<Message>> producerStreams[] = recordStream.mapValues(new ValueMapper<Message, List<Message>>() {
        @Override
        public List<Message> apply(final Message message) {
          return consumerRecordHandler.process(message);
        }
    }).*someFunction*()

Here, the record read is Message; After processing it returns a list of Message. How can I divide this list to two producer streams? Any help will be appreciated.

score 18 · Accepted Answer · edited Feb 17 '21 at 16:41

18

I am not sure if I understand the question correctly, and I also don't understand the answer from @Abhishek :(

If you have an input stream, and you want to get zero, one, or more output records per input records, you would apply a flatMap() or flatMapValues() (depending if you want to modify the key or not).

You are also asking about "How can I divide this list to two producer streams?" If you mean to split one stream into multiple, you can use branch().

For more details, I refer to the docs: https://docs.confluent.io/platform/current/streams/developer-guide/dsl-api.html#stateless-transformations

edited Feb 17 '21 at 16:41

Benissimo

1,010
9
14

answered Jun 02 '17 at 00:08

Matthias J. Sax

59,682
7
117
137

@user2538255 Feel free to follow up if my answer is unclear. – Matthias J. Sax Jun 02 '17 at 00:10
Thats exactly what I am doing. After some googling on Abhishek's answer, I landed on this example https://github.com/confluentinc/examples/blob/kafka-0.10.0.0-cp-3.0.0/kafka-streams/src/test/java/io/confluent/examples/streams/WordCountLambdaIntegrationTest.java – user2538255 Jun 02 '17 at 08:49
@MatthiasJ.Sax is there a way to achieve some kind of transactional flat mapping? If one records processes N as a result of a flat map, I want all N or 0 in the outbound topic. – Никита Михайлов Jul 11 '19 at 09:35
1

If you enable exactly once processing, you get what you ask :) – Matthias J. Sax Jul 11 '19 at 16:05
@MatthiasJ.Sax - Is the example of WordCountLambdaIntegrationTest.java mentioned by @ user2538255 available in the latest version of confluent examples? – adbdkb May 26 '21 at 14:04
https://github.com/confluentinc/kafka-streams-examples/blob/6.1.1-post/src/test/java/io/confluent/examples/streams/WordCountLambdaIntegrationTest.java – Matthias J. Sax May 26 '21 at 20:08

score 3 · Answer 2 · answered May 31 '17 at 08:44

What's your key (type) ? I am guessing its not String. After executing the mapValues you'll have this - KStream<K,List<Message>>. If K is not String then someFunction() can be a map which will convert K into String (if its is, you already have the result) and leave the List<Message> (the value) untouched since that's your intended end result

Kafka Streams: one record to multiple records

2 Answers2