0

We have a usecase where we need to consume the data from one of the Kafka Cluster on some Topic like 'daily-data' and we need to filter them and send it to a Azure Event Hub topic say 'our-daily-data'. By Default, kafka streams doesn't support posting the messages on some topic to different cluster

Is there any workaround or pointers would be helpful.?

satya
  • 109
  • 2
  • 14
  • One of the approach we used is creating a new producer for Kafka enabled Event hub and sending the filtered messages to the eventHub. However we see a single timeout exception occurring for every 5 messages. – satya Feb 27 '20 at 13:11
  • Able to solve using a separate a kafkaProducer – satya Feb 27 '20 at 17:16

1 Answers1

0

You can run MirrorMaker for your scenario. Please see detils here - https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-kafka-mirror-maker-tutorial

enter image description here

Serkant Karaca
  • 1,896
  • 9
  • 8
  • we cannot use a mirror maker because: 1) Size of the messages on Kafka Size is 10MB where as the EventHub supports 1MB. 2) kafka messages load is around 25M+ records daily and Messages that are related to our application could be 250K at most in that 25M. – satya Feb 27 '20 at 07:14
  • For large message scenarios, we recommend to store messages in another store like Azure Storage and send identifier of the messages such as URIs to Event Hubs. – Serkant Karaca Mar 02 '20 at 17:52
  • Thanks. I am able to do using Kafka Producer to EventHub. I transformed the incoming message to a smaller size and posted to EventHub. – satya Apr 17 '20 at 10:55