1

Using kafka processor API( not DSL) to read from a source topic and write to target topic it works fine for a single kafka cluster setup (that is if both source and target topics reside on the same cluster) , but when source and target topics reside on different kafka clusters I am getting NullPointerException for the target processor context 

Topology
topology.addSource("mySource", "SourceTopic");
topology.addProcessor("SourceStreamProcessor",()->new SourceStreamProcessor(), "mySource");         

topology.addProcessor("TargetProcessor",()->new TargetProcessor(), "Target");
topology.addSink("sink1","OUTPUT_TOPIC1","TargetProcessor");
topology.addSink("sink2","OUTPUT_TOPIC2","TargetProcessor");


Properties sourceProcessorProps = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "SourceStreamProcessor"); // Kafka Cluster 1
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dev_Cluser_xx.org:9092");

Properties targetProcessorProps = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "targetStreamProcessor"); // Kafka Cluster 2
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "test_Cluser_xx.org:9092");

How can we use kafka streams processor API to write from one topic in one cluster to another topic in a different cluster ?

Maazen
  • 107
  • 1
  • 14
  • How do you build the `KafkaStreams` instance with two `Properties` objects, when its constructor takes only one `Properties` parameter? https://kafka.apache.org/20/javadoc/org/apache/kafka/streams/KafkaStreams.html#KafkaStreams-org.apache.kafka.streams.Topology-java.util.Properties- – JavaTechnical Jun 19 '20 at 05:33

1 Answers1

1

It is not supported by Kafka Streams to read from one Kafka Cluster and to write to the other Kafka Cluster.

You can process messages within one cluster and than using Mirror Maker copy it to the other.

Bartosz Wardziński
  • 6,185
  • 1
  • 19
  • 30