1

I am trying to design a system that can be safely used to share messages between processes on the same system using ZeroMQ.

I have chosen to use a central broker that binds to a particular port. All clients would bind to the same port and publish to this using different topics.

My Requirements

  1. I have a network of multiple consumers and producers that talk to each other in a bi-directional flow. .i.e a consumer can occasionally act as a producer and vice versa.
  2. Given two services A&B, there are messages that are unique to A&B.
  3. Given services A,B...Z there are messages that are not unique to A&B .i.e there are messages that are consumed by A&B, but also by another service C.

My Approach

  1. Have central broker that connects to a specified port. All producers publish to this broker through this ZMQ context and subscribe from the same context. The socket type varies one is a PUB socket, other is a SUB socket. So essentially each service has one ZMQ context but two different sockets.

  2. Service A&B can communicate to each other using the same broker, for unique messages I use a unique topic, lets say topic AB.

  3. For non-unique messages consumed by multiple services, I have one topic say common.

My thought process

  • Using a single broker makes it simpler, as opposed to this I would be looking at each service handling multiple ZMQ contexts, .i.e one context for each service it wants to talk to. Thus creating an overhead.
  • Broker solves one problem of routing thus very important, but at the same time its dumb, and resilient.
  • Unless the broker goes down, one service going down will not affect the other services.

My questions

  • Can the broker be made intelligent to monitor for crashes, queue when clients are down etc. I understand ZMQ the solves queuing at the subscriber level, hence freeing up the publisher, but with a broker, can it be done here? This is to add a layer of resilience at the broker level in case the clients go down.

  • Are the messages duplicated in this design?.

  • When looking at the common topic .i.e messages which are not unique to any service but are consumed by multiple services. Is there a way to add sub-topics / sub filtering?

Also, For communicating unique messages between services, would ZMQ push pull pattern help more than PUB SUB?

For example if there are 1000 messages on the common topic, I have sink that consumes to all messages.

Lets say a service C needs only 10 messages of these 1000, I don't want to filter all 1000 messages on the subscriber .i.e C, I would rather send only the 10 messages to C.

I don't see any way of doing this, also all my senses say this should not be technically possible as well. But want to know if there is any way to do it , I may have missed something.

The only way I see possible is:

  1. Send on different topics, on the sink, consume all relevant topics and multiplex them.
  2. Filter all messages on the common topic on each subscriber to get what each subscriber wants.

Flow

Abhilash Gopalakrishna
  • 910
  • 2
  • 22
  • 52
  • If you want a single broker with queuing and other resiliency features, why not just run an actual messaging service rather than trying to design your own? Stand up redis or rabbitmq or something and use that, because it sounds like zeromq isn't a great match for your particular use case. – larsks Jan 26 '22 at 12:26

1 Answers1

0

Resilience

Regarding your first question about a resilient broker; you can set options on sockets so that the endpoints ping-pong each other, and notify their endpoint through the socket file descriptor if these stop being exchanged.

The idea is that if one processes dies or the machine hosting it stops working, the process at the other endpoint gets told that comms has stopped. The surviving process then has to decide what to do with the messages its got, and there's general question about what to do about the dead process. But, knowing that something has gone wrong is a good start.

Duplication

Messages in patterns like PUB SUB are duplicated; that's the idea, even if everything is in the same machine. So arguably it makes little difference if C were to send its "common" msesages to B and the sink via separate point to point PAIR sockets.

Sub Topics

I'm not sure there's a way in ZMQ to have sub topics, but it would be easy to implement this oneself in the process at the receiving endpoints. For example, all SUBs could be subscribed to a topic "MyMajorTopic", and the first part of the message could be reserved for a sub topic string (or other identifier) that the processes has to check before deciding what to do with the rest of the message.

PUSH PULL Instead of PUB SUB

The PUSH PULL pattern is more about sending messages to the "next available" consumer. This does not allow you to have strict rationing of messages between PULL consumers, because it all depends on the rate at which the PULL consumers deal with the messages they receive and call msg_recv() again.

So, C might need only 10 in 1000, but AFAIK there's no way to influence a PUSH PULL pattern so that C gets only 10 messages in 1000. You'd have to create this rationing in your processes, and likely use 2 different sockets from the message producer (one to C, the other to your sink).

Overall

On the whole, it's probably best to keep things simple. That can be more things, but less complex things.

For example, you have Common topic messages coming from A, B and C to Sink and B. I'd treat that as one comms multipex, i.e. have sockets (and possibly a broker process) just for this Common topic. That broker then ends up being very simple. I'd then have a different set of sockets (and possibly a broker) for the comms between A, C towards B; that too is then very simple.

bazza
  • 7,580
  • 15
  • 22