0

I have a Java web-service that I am going to reimplement from scratch in Scala. I have an actor-based design for the new code, with around 10-20 actors. One of the use-cases has a flow like this:

Actor A gets a message a, creates tens of b messages to be handled by Actor B (possibly multiple instances, for load balancing), producing multiple c messages for Actor C, and so on.

In the scenario above, one message a could lead to a few thousand messages being sent back and forth, but I don't expect more than a handful of a messages a day (yes, it is not a busy service at the moment).

I have the following requirements:

  1. Messages should not be lost or repeated. I mean if the system is restarted in the middle of processing b messages, the unprocessed ones should be picked up after restart. On the other hand, the processed ones should not be taken again (these messages will in the end start some big computation, and repeating them is costly).
  2. It should be easily extensible. I mean in the future, I may want to add some other components to the system that can read all the communication (or parts of it) and for example make a log of what has happened, or count how many b messages were processed, or do something new with the b messages (next to what is already happening), etc. Note that these "components" could be independent applications written in other languages.

I am new to message bus technologies, but from what I have read, these requirements sound to me like what "message buses" offer, like RabbitMQ, Kafka, Kestrel, but I also see that akka also offers some means for persistence. My problem is, given the huge range of possibilities, I am lost which technology to use. I read that something like Kafka is probably an overkill for my application. But I am also not sure if akka persistence answers my two requirements (especially the extensibility).

My question is: Should I go for an enterprise message bus? Something like Kafka? Or something like akka persistence will do? Or would it be just faster and more appropriate if I implement something myself (with support for, say, AMQP to allow extensibility)?

Of course, specific technology suggestions are also welcome if you know of something that fits this purpose.

Mahdi
  • 1,778
  • 1
  • 21
  • 35
  • Akka by itself supports _at-most-once_ delivery (meaning that messages can be lost). If you add in Akka Persistence, you can achieve _at-least-once_ delivery (meaning that you can get duplicates) by using the `AtLeastOnceDelivery` trait. What you want, however, is _exactly-once_ delivery. That is not supported by Akka out of the box, and there are those who say it cannot be achieved. See for example https://dzone.com/articles/you-cannot-have-exactly-once. You can get close, though, but you have to work for it. – Mario Camou Jul 23 '15 at 12:34
  • Thanks @MarioCamou for your comment. In fact, I have seen that post. So, shall I take your suggestion as "build it yourself"? – Mahdi Jul 23 '15 at 13:08
  • Yes, I would personally start with at-least-once delivery and then deal with resends and out-of-order messages. Note that even if you use Kafka or a message queue you have to consider the fact that the message queue itself might go down (and with Akka Persistence you also have to think about what if your persistence provider goes down). – Mario Camou Jul 23 '15 at 13:25

1 Answers1

0

A Message Bus (typically called Message Brokers) like RabbitMQ can handle "out of the box" all of the messaging mechanisms you describe in your question. Specifically:

RabbitMQ has the ability "Out of the Box":

  • To deliver messages without repeating the message.
  • To extend the system and add logging and have statistics like you describe.
joshuad2
  • 309
  • 1
  • 4
  • Thanks for your answer @joshuad2 . Question is, how much is its overhead? I mean considering the fact that my system is not very large-scale, would you recommend to employ something like RabbitMQ? For example, considering the maintenance and administration costs. – Mahdi Jul 23 '15 at 15:51
  • RabbitMQ is very light weight in comparison to other Message Brokers such as Kafka. The size of the executable is quite small and it will only use what you put into the Broker. Therefore, if you don't have many messages in the queues at any one time the size and overhead is quite small, but the benefits are high considering your routing needs. – joshuad2 Jul 23 '15 at 16:43
  • OK. I accept your answer, although it will sure take time until I can actually implement and verify it! ;-) – Mahdi Jul 27 '15 at 09:09