Kafka on Masstransit

Question

I am studying the possibility to use Kafka with Masstransit on our Information System. Masstransit documentation says Kafka may be used but as a Rider; As the usage of a transport is mandatory we must use another tool (as ActiveMQ, RabbitMQ) in addition of Kafka with Masstransit.

What does transport do that can not be done by Kafka, that prevent exclusive usage of Kafka?

@OneCricketeer, in this post i assume my precedent question was too vague, so i focus on masstransit, not on Saga globally; but if you asked me this question this sounds you understood what was my problem. Sorry for precision lack, but is not easy to deal with a problem that is not clearly identified — user1958187, Aug 25 '21 at 12:53

score 3 · Answer 1 · answered Aug 25 '21 at 11:02

3

MassTransit is designed with publish-subscribe with competing consumers in mind, which is the default pattern when using a message broker.

Riders were introduced, as mentioned in the docs, as a gateway between event streaming and pub-sub. So, the main purpose of riders is to support the scenario when you need to consume a message from a rider and publish it to the bus. Even producers are optional for riders, and they are implementation-specific. As event streaming infrastructure like Kafka doesn't even support competing consumers, it makes little sense to make something like Kafka transport, as it introduces a heavy impedance mismatch between MassTransit concepts and what Kafka does.

You can still only use Kafka with MassTransit, but you'd need to use the in-memory transport to configure and start the bus.

answered Aug 25 '21 at 11:02

Alexey Zimarev

17,944
2
55
83

Thanks for your answer; you say that Kafka does not support competing consumers... but it does inside a consumer group, doesnt' it? – user1958187 Aug 25 '21 at 13:03
1

They are not competing. You can't have more than one consumer per partition within a group, so there's no "competition". – Alexey Zimarev Aug 25 '21 at 13:11
1

Indeed, so it may be a limitation of Kafka – user1958187 Aug 25 '21 at 13:12
@user1958187 sure, if by "limitation" you mean "powerful design choice". The competing consumers pattern destroys message ordering. – MattDavey Sep 19 '21 at 19:51

Kafka on Masstransit

1 Answers1