0

I'm trying to evaluate different pub/sub messaging protocols on their ability to horizontally scale without producing unnecessary cross chatter.

My architecture will have NodeJS servers with web socket clients connected. I plan on using a consistent hashing based router to direct clients to servers based off of the topics they're interested in subscribing to. This would mean that for a given topic, only a subset of servers will have clients subscribing to that topic. Messages will then be published to a pub/sub broker, which would be responsible for fanning out that data to servers that have subscribers.

The situation I want to avoid is one in which every broker receives every request, and the network becomes saturated. This is a clear issue with scaling Redis Pub/Sub. Adding servers shouldn't create an n squares' problem.

The number of clients on the pub/sub protocol would be the number of servers. Ideally, each server would be able to have a local broker to fan out data efficiently to multiple NodeJS processes, as to avoid unnecessary network bandwidth. In most cases, for a given topic, all subscribers would be on that same server.

What pub/sub protocols offer this sort of topic based data propagation?

The protocols I'm evaluating are: MQTT, RabbitMQ, ZMQ, nanomsg. This isn't inclusive, and SAAS options are acceptable.

The quality assurance constraints are easy. At most once, or at least once are both adequate. Acknowledgment isn't important. Event order isn't important. We're looking for fire and forget, with an emphasis on horizontal scalability.

user1363145
  • 107
  • 1
  • 2
  • 6
  • Note - horizontal upscaling in your case needs all the nodes to know the same information. If you intend to mitigate the info using queues or other RPC, you MUST incroduce the n-square problem. Another way is to either use something that uses n-square solution (like in-memory grid), or push the problem down by using a distributable persistance layer (like ant no-sql databse). – Henry Aloni Sep 27 '15 at 07:40
  • Two comments: 1) Do you really ask for _protocols_ to avoid N^2? I would assume you are more interested in protocol broker implementations. 2) Is there any restriction on the communication pattern? If all Sub's subscribe to all messages issued by all pub's, then it feels to me like an inherent N^2 issue. However, if each sub has its own topic, you could e.g. partition on topic. – Stefan Vaillant Sep 28 '15 at 06:38
  • Adding Lightstreamer to the list. It implements what we call "publish-on-demand" paradigm, where each server notifies its data feed that it can start publishing data on a given topic only when there is at least one subscriber connected. – Alessandro Alinone Sep 28 '15 at 09:00
  • 1) I'm most concerned about PUB-level filtering, which seems like a key component necessary to avoid an N^2 architecture issue. 2) No. 3) Say for instance there are 1000 topics, and 10 servers... maybe each server subscribes to 100 topics. As long as they're not receiving messages from the other 900 topics, we're golden – user1363145 Sep 29 '15 at 14:12

1 Answers1

0

First, let me address a risk of mis-understanding

In many cases, similar words do not mean the same thing. The more the abbreviations.

Having that said, let me review a PUB/SUB terminus technicus.

Martin SUSTRIK's and Pieter HINTJENS' team in imatix & 250bpm have developed a few smart messaging frameworks over the past decades, so these guys know a lot about the architecture benefits, constraints and implementation compromises.

That said helps me to state that these fathers, who introduced grounds of the modern messaging, do not consider PUB/SUB to be a protocol.

It is, at least in nanomsg & ZeroMQ, rather a smart Distributed Scaleability-focused Formal Communication Pattern -- i.e. a behaviour emulated by all involved parties.

Both ZeroMQ and nanomsg are broker-less.

In this sense, asking "what protocols" does not have solid grounds.

Let's start from the "data propagation" side

In initial ZeroMQ implementations PUB had no other choice but distribute all messages to all SUB-s that were in a connected-state. Pieter HINTJENS explained numerous times this decision that actual subscription-based filtering was performed on SUB-side ( distributed in 1:all-connected manner ).

It came much later to implement PUB-side subscription based filtering and you may check revisions history to find since which version this started to avoid 1:all-connected broadcasts of data.

Similarly, you may check the nanomsg remarks from Martin SUSTRIK, who gave many indepth posts on performance improvements designed in his fabulous nanomsg project.

Scaleability as a priority No.1

If Scaleability is the focus of your post and if it were a serious Project, my question number one would be what is the quantitative metric for comparing feasible candidates according to such Project goal - i.e. what is the feasibility translated into a utility function to score candidates to compare all the parallel attributes your Project is interested in?

user3666197
  • 1
  • 6
  • 50
  • 92
  • The goal is to handle millions of concurrent websocket clients connected to a distributed network of NodeJS servers, and to be be able to send messages to all users interested in given topic internally, between servers. My criteria are based on pragmatism. All candidates need to have **`PUB`** based subscription filtering to prevent an n squares' network saturation dilemma when adding new servers. After that, I mainly care about NodeJS support, DevOP's simplicity (preferably there's an automated solution, SAAS/docker/AWS add on etc), and scalability. – user1363145 Sep 28 '15 at 06:22
  • The more if you go into production-grade stability/scaleability/fault-resilience review your principal architecture singularities. ZeroMQ / nanomsg are **broker-less, i.e. one SPOF less for the fault-resilience + no central data-flow singularity**. Also the **performance scaling** ( load-balancing parts may help your architecture avoid adverse scaling-related issues ) **is almost linear**, so referring to parabolic scaling does not make sense here. **Network saturation is out of game**, should you go in for **PUB-side filtering**. A smart principal architecture is key. Or did I miss something? – user3666197 Sep 28 '15 at 14:43
  • That's pretty accurate. PUB-side filtering lets me avoid network saturation, assuming the rest is architected well. – user1363145 Sep 29 '15 at 14:15
  • It looks like, from what I've researched, that nanomsg might be the best candidate in that it implements PUB-side filter, has better capabilities for handling larger number of subscription patterns using a radix tree, among other benefits. – user1363145 Sep 29 '15 at 14:25
  • Well, as far as the "millions" of subscriptions are concerned, my experience yields to more important factors, that can help solve this "less-interesting" attribute. These are non-blocking, low-latency, massive-parallel, self-autodiagnosed, (N+M) self-resilient messaging planes, where these millions simply lose their "illusion of a Big issue". Good luck with nanomsg. May opt to try to design your solution as a framework-agnostic & benefit an option to snap-in a 0MQ, in case nanomsg-0.6-beta or previous releases make your Project into showstopper issues. I love smart & efficient tools, both are – user3666197 Sep 29 '15 at 14:56