7

I need to read from a specific Kafka topic, do a VERY short processing on the message and pass it on to a different Kafka cluster.

Currently, I'm using a consumer that's also a producer on the other kafka server.

However, the streaming API supposedly offers a more light-weight high-throughput option.

So the questions are:

  • Assuming my processing code doesn't require much horse power, is the streaming API better?
  • Does the streaming APi support writing to a different Kafka cluster?
  • What are the Streaming API cons comparing to the Consumer API?
Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
sternr
  • 6,216
  • 9
  • 39
  • 63

1 Answers1

3

Unfortunately KafkaStreams doesn't currently support writing to a different Kafka cluster.

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
Damian Guy
  • 131
  • 3
  • 1
    As a work around, you could use `#process()` and use an extra `KafkaProducer` instance there. Cf. http://stackoverflow.com/questions/41796207/dynamically-connecting-a-kafka-input-stream-to-multiple-output-streams – Matthias J. Sax Feb 27 '17 at 18:28