0

I think my question is simple but I am unable to figure it out,

#1: I have a db table that has different vendors with their kafka topic entry

VENDOR     TOPIC
-----------------------
Paytm     PaytmTestTopic
Phonepe   PhonePeTestTopic
GPay      GpayTestTopic
Roger     RogerTestTopic

there are more entries.

#2: I have a KafkaStream Listening to a transaction topic.

    final StreamsBuilder builder = new StreamsBuilder();
    final KStream<String, String> stream = builder.stream("payment_txn_topic");

#3: stream has a deserializer, factory and chunks serializer to transform the output

    stream.transformValues(eventDeserializer)
            .flatMapValues(chunkFactory)
            .transformValues(chunkSerializer)
            .to("");

4: I need to have the topic name dynamically extracted I know how to use TopicNameExtractor to get the topic name and pass it to .to() for output.

But here is my use-case:

My Event will be somewhat {"modeList":[{"mode":"Paytm"}, {"mode":"Roger"}]}

so now I need the extractor so that it can subscribe both "PaytmTestTopic" and "RogerTestTopic" i.e. depending the modes in event as list.

Q1: As we can return only one topic name from TopicNameExtractor, how can I do it for multiple met conditions in single event ?

Q2: Anyway to use Extractor based on transformed data to be sent ? as TopicNameExtractor works on received event.

Q3: Can I use dynamic branches and/or KTable to achieve it?

Any help will be much appreciated, Thanks in advance.

Bilal Siddiqui
  • 3,579
  • 1
  • 13
  • 20
  • Seen this? https://stackoverflow.com/questions/48950580/kafka-streams-send-on-different-topics-depending-on-streams-data or https://stackoverflow.com/questions/57979679/send-message-to-topic-based-on-message-key-using-kafka-streams – OneCricketeer Jun 03 '21 at 01:30
  • @OneCricketeer Yes I saw those. But in my case the vendors data is a heavy list and I want a dynamic approach instead of hard coding. that is for whatever vendors modelist contains the to need to be constructed. – Bilal Siddiqui Jun 03 '21 at 03:35
  • @OneCricketeer and as the event message contains multiple modes so how can I map to a dynamic branches ? I saw your comment on 2nd one which has a dummy list prior to the event. – Bilal Siddiqui Jun 03 '21 at 05:19
  • The "dummy list" can come from anywhere, like a file or an external database query like you've shown here – OneCricketeer Jun 08 '21 at 21:26

0 Answers0