0

Can I use processor api in Kafka Streams and read data from a topic in one kafka cluster and pipe data into a kafka topic in ANOTHER kafka cluster?

Is that do-able thing using processor api? I know, its not possible with Stream DSL!

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Raja
  • 513
  • 5
  • 18
  • 1
    It could be possible if you manually create a Producer object yourself within the Processor, but as mentioned, MirrorMaker is the official builtin tool for this – OneCricketeer Sep 16 '18 at 14:17
  • Duplicate: https://stackoverflow.com/questions/45847690/kafka-streams-connecting-to-multiple-clusters – Matthias J. Sax Sep 16 '18 at 18:26
  • If you create your own instance of `KafkaProducer` you will need to go sync writes to guarantee at-least-once processing. Thus, performance is expected to drop. It's not a recommended pattern. – Matthias J. Sax Sep 16 '18 at 18:27

1 Answers1

3

This is not possible with Kafka Streams. It only works against a single cluster.

To copy data across clusters you need to either use MirrorMaker or one of the available replicator tools, for example a Kafka source connector.

Mickael Maison
  • 25,067
  • 7
  • 71
  • 68
  • Thanks for the response. Is there a way with mirror maker, i can configure the a different topic name in the destination cluster. By default, the script is expecting the topic name to be same in both clusters! Is it possible to change the name in destination cluster ? – Raja Sep 17 '18 at 08:43
  • By default, it's not possible with MirrorMaker so you'll have to implement it yourself – Mickael Maison Sep 17 '18 at 11:18
  • ok. Thanks. One question, Can i write a application using processor api, and stream msgs to another cluster ? any link example programs for processor api you can help with ? – Raja Sep 17 '18 at 11:32