Message Broker - Multivendor Architecture in .NET Core

Question

I am currently building a system following the "competing consumer pattern" in .Net Core. Thus, I put a JobRequest-Message on a queue on which multiple consumers (workers) are subscribed to. One of the consumer receives the message an does the actual processing of the job.

I also have the requirement to build the system "vendor independent". Meaning that I need to implement the system in a way, that allows me to easily switch between message broker vendors, for instance: RabbitMq, Azure Service Bus, Amazon SQS, (Kafka).

We are currently discussing the best option to archieve that and came up with following alternatives:

1. Interfaces and different implementations for each vendor

I create interfaces for the various features I need for my system and use the client libraries of the vendors to implement that interfaces. The drawback is of course, that I need to write an implementation for each vendor.

2. using AMQP 1.0 Standard Client Libraries

I stick to the AMQP standard and use AMQP standard libraries (https://github.com/Azure/amqpnetlite) for my implemenation. All AMQP based message brokers should then work with just one implementation. Drawback is, that I am dependent on AMQP and also the AMQP version (RabbitMQ currently using 0.9, Kafka is not using AMQP).

3. using a service bus on top that already supports different brokers

Like NServiceBus or MassTransit (open source). That would add an additional dependency to my system and increases its footprint. NServiceBus is also not free and it would somehow just shift the "vendor depencency" to the service bus.

Number one quality goal is to build a reliable and resilient system (> 100.000 Jobrequests per day).

Please share your thoughts :)

Answer 1

I must admit that I am biased, because I am the author of Rebus. Nevertheless, here's my recommendation:

Go with option 3!

All of the mentioned service bus libraries are fairly un-intrusive, meaning that the bulk of the code you end up writing will take a form that looks like this (examples taken from Rebus, but the other two mentioned libraries are very similar):

await bus.Send(command);

for sending a command, or

await bus.Publish(evt);

for publishing an event, where bus is an interface (IBus in this case), and like this:

public class MyCommandHandler : IHandleMessages<MyCommand>
{
    public async Task Handle(MyCommand command)
    {
        // do stuff with the command here
    }
}

for handling messages (regardless of whether they were sent or published).

This means that you will get ALL THE BENEFIT of the code portability and independence from vendors delivered by the library, with a very small impact on the layout of your code.

Should you ever decide to implement your own messaging code, it should not take too much work to replace the use of the service bus library with your own messaging code: Creating your own IBus interface and your own IHandleMessages<TMessage> interface would be enough to remove the dependency entirely in this case.

Just my two cents.