0

I am looking for a message queue which would replicate messages across a cluster of servers. I am aware that this will cause a performance hit, but that's what the requirements are - message persistence is very important.

The replication can be asynchronous, but it should be there - if there's a large backlog of messages waiting for processing, they shouldn't be lost.

So far I didn't manage to find anything from the well-known MQs. HornetQ for example supported message replication in 2.0 but in 2.2 it seems to be removed. RabbitMQ doesn't replicate messages at all, etc.

Is there anything out there that could meet my requirements?

adamw
  • 8,038
  • 4
  • 28
  • 32
  • So you are looking for a form of *guaranteed* messaging, where you are protected at the broker side from a power outage or a disk failure? So in other words, once the broker has the message, you can pretty much assume that shortly thereafter, given async, you can lose both the publisher and that broker box, and still recover the message? – sdg Nov 22 '11 at 14:02
  • Yes, I'd like the message replicated to another machine. – adamw Nov 22 '11 at 14:23

3 Answers3

2

There are at least three ways of tackling this that come to mind, depending upon how robust you need the solution to be.

One: pick any messaging tech, then replicate your disk-storage. Using something like DRBD you can have the file-backed storage copied to another machine under the covers. If your primary box dies, you should be able to restart on your second machine from the replicated files.

Two: Keep looking. There are various commercial systems that definitely do this, two such (no financial benefit on my part) are Informatica Ultra Messaging (formerly 29West) and Solace. These are commonly used in the financial community.

Three: build your own. ZeroMQ is one such toolkit that you could use to roll-your-own system from pre-built messaging blocks. Even a system that does not officially support it could fairly easily be configured to publish all messages to two queues. Your reader would have to drain both somehow, so this may well be a non-starter, but possible in any case.

Overall: do test your performance assumptions, as all of these will have various performance implications in various scenarios.

sdg
  • 4,645
  • 3
  • 32
  • 26
  • Thanks! :) I'll sure consider the "build your own" scenario in my fre time. Meanwhile I added a thin layer on top of MongoDB, which when used in a replica set works as I'd like it to. – adamw Dec 02 '11 at 16:17
0

new Kafka 0.8.1 offers replication!

abelard2008
  • 1,984
  • 1
  • 20
  • 35
0

Amazon SQS is designed with this very thing in mind, but because of the consistency model (which is a part of messaging anyway), you're responsible for de-duplicating messages on the consumer side. Granted, SQS maybe somewhat slow and the costs can add up for lots of messages, but if you want to guarantee that no messages are lost, then it's a pretty solid way to go.

Jonathan Oliver
  • 5,207
  • 32
  • 31