0

We have an application that forwards data to MapR Streams (Kafka). Problem is if the network link goes down then data is lost as the application itself doesn't keep a history. We are looking at installing a message bus of some sort on the same VM so the application can buffer data in the local message bus. Another process will read from that message bus and forward to MapR Streams. Before I do all that I'm wondering if there is such a thing as a message forwarder? I know Splunk has this concept but it is specific to Splunk. This would be a process that runs on the same server as the application, receives messages, forwards them to another messagebus and stores them locally if that fails. Once sent a message would be deleted.

MikeKulls
  • 336
  • 1
  • 2
  • 16
  • 1
    Wouldn't you run into the same issue if the message bus dies? I understand it is in the same VM but it could still fail. Is the application working as a kafka producer? – Tux_DEV_NULL Jul 12 '19 at 12:46
  • That is true but it is much less likely to die than a network link of potentially 100s of kms. If it does die, we have redundant VMs for the application so we could either forward to the message bus on the other VM or consider that a complete failure of the application and fail over to the other VM. The application collects data from routers via CLI and currently writes the data to files. The files are picked up via SFTP and pushed to MapR Streams. We are looking to eliminate the files without losing the fact they act as a buffer. – MikeKulls Jul 17 '19 at 05:15

0 Answers0