0

I am using Kafka streaming and i have a doubt.

My code is 

final KStream<String, Entity> inStream = builder.stream(TOPIC);
inStream.map((key, entity) -> {
    ....
    return new KeyValue<>(key, entity);
}).to(NEW_TOPIC);`

Value of NEW_TOPIC is present in entity object. My problem is how to i extract the value of this NEW_TOPIC from entity in case of multiple task running.

My problems drill down to the fact that if there are multiple task, would kafka-streaming process the incoming message till end (by calling the to() method to push it back to new kafka topic) and then pull new message from incoming topic? If this is the behavior, i can store this value in a local /final variable to use it later. If this is not the behavior then i need to use some other way.

pwolaq
  • 6,343
  • 19
  • 45
Wadi
  • 29
  • 5
  • yes, depending on the type of entity object, there would be different topics. This info would be part of entity. – Wadi Dec 27 '18 at 12:25
  • Not possible , read https://issues.apache.org/jira/browse/KAFKA-4936 – Himeshgiri gosvami Dec 27 '18 at 12:27
  • follow this link https://stackoverflow.com/a/41799272/10722561 you can create a producer and produce message put it may be slow. – Himeshgiri gosvami Dec 27 '18 at 12:32
  • @Himeshgoswami the link you shared is for different issue. As i understand, that is for extracting topic details within the payload, in my case i am specifically providing which topic it has to go. – Wadi Dec 27 '18 at 12:38
  • Could you describe how you do _"in my case i am specifically providing which topic it has to go."_? – Jacek Laskowski Dec 27 '18 at 20:56
  • org.apache.kafka.streams.processor.TopicNameExtractor solved my problem. – Wadi Jan 15 '19 at 12:01

0 Answers0