1

Consider the following:

  • a set of 3 logical services: S1, S2 and S3
  • two instances of each service are running, so we have the following processes: S1P1, S1P2, S2P1, S2P2, S3P1, S3P2
  • a ZeroMQ broker running in a single process and reachable by all service processes

A logical service, let's say S1, publishes a message M1 that is of interest to logical services S2 and S3. Only one process of each logical service must receive M1, so let's say S2P1 and S3P2.

I have tried the following, but without success:

  • broker thread 1 is running a XSUB/XPUB proxy
  • broker thread 2 is running a ROUTER/DEALER proxy with the ROUTER connected to the XPUB socket and subscribed to everything (for logical S1)
  • broker thread 3 is running a ROUTER/DEALER proxy with the ROUTER connected to the XPUB socket and subscribed to everything (for logical S2)
  • broker thread 4 is running a ROUTER/DEALER proxy with the ROUTER connected to the XPUB socket and subscribed to everything (for logical S3)
  • each logical service process is running a REP socket thread connected to the broker DEALER socket

I figured that the XSUB/XPUB proxy would give me publish/subscribe semantics and that the ROUTER/DEALER proxies would introduce a competition between the REP sockets for the messages sent by the XSUB/XPUB proxy.

How can I combine ZeroMQ sockets to accomplish this?

Update1

I know "without success" isn't helpful, I've tried different configurations and got different errors. The latest configuration I tried is the following:

(XSUB proxy=> XPUB) => (SUB copyLoop=> REQ) => (ROUTER proxy=> DEALER) => REP

The copyLoop goes like this:

public void start() {
    context = ZMQ.context(1);

    subSocket = context.socket(ZMQ.SUB);
    subSocket.connect(subSocketUrl);
    subSocket.subscribe("".getBytes());

    reqSocket = context.socket(ZMQ.REQ);
    reqSocket.connect(reqSocketUrl);

    while (!Thread.currentThread().isInterrupted()) {
        final Message msg = receiveNextMessage();
        resendMessage(msg);
    }
}

private Message receiveNextMessage() {
    final String header = subSocket.recvStr();
    final String entity = subSocket.recvStr();

    return new Message(header, entity);
}

private void resendMessage(Message msg) {
    reqSocket.sendMore(msg.getKey());
    reqSocket.send(msg.getData(), 0);
}

The exception I get is the following:

java.lang.IllegalStateException: Cannot send another request
    at zmq.Req.xsend(Req.java:51) ~[jeromq-0.3.4.jar:na]
    at zmq.SocketBase.send(SocketBase.java:613) ~[jeromq-0.3.4.jar:na]
    at org.zeromq.ZMQ$Socket.send(ZMQ.java:1206) ~[jeromq-0.3.4.jar:na]
    at org.zeromq.ZMQ$Socket.sendMore(ZMQ.java:1189) ~[jeromq-0.3.4.jar:na]
    at com.xyz.messaging.zeromq.SubReqProxyConnector.resendMessage(SubReqProxyConnector.java:47) ~[classes/:na]
    at com.xyz.messaging.zeromq.SubReqProxyConnector.start(SubReqProxyConnector.java:35) ~[classes/:na]

I'm running JeroMQ 0.3.4, Oracle Java 8 JVM and Windows 7.

Spiff
  • 2,266
  • 23
  • 36
  • You might want to know that **`ZeroMQ` is broker-less by-design** – user3666197 Mar 11 '16 at 10:42
  • When you say "without success" - what issues are you running into? I'll suggest an alternate (but similar) architecture in an answer, but it may not be helpful depending on where you're running into issues. – Jason Mar 11 '16 at 14:44
  • @Jason You're absolutely right that it wasn't clear - it's because I tried different combinations and got different exceptions. I've updated the question to reflect the latest configuration. – Spiff Mar 11 '16 at 15:04
  • @user3666197 Yes I'm aware of that, I'm trying to determine whether I can build a bare-bones broker with ZeroMQ in order to avoid installing a new component like RabbitMQ in a customer's environment. – Spiff Mar 11 '16 at 15:06
  • @user3666197 I also find it odd that ZeroMQ is marketed as broker-less by design, yet one of the first patterns that are presented in the guide is a pub/sub broker. – Spiff Mar 11 '16 at 15:07
  • 1
    I wouldn't call ZMQ broker-less, I'd say ZMQ is a messaging library at a lower level than the broker. You design your topology based on your messaging requirements, and if it requires a broker, you build one, if it doesn't you don't. – Jason Mar 11 '16 at 15:12
  • Oh, and thanks for the clarification on what's going wrong, that helps me immensely – Jason Mar 11 '16 at 15:13

2 Answers2

3

You seem to be adding in some complexity with your ROUTER connection - you should be able to do everything connected directly to your publisher.

The error you're currently running into is that REQ sockets have a strict message ordering pattern - you are not allowed to send() twice in a row, you must send/receive/send/receive/etc (likewise, REP sockets must receive/send/receive/send/etc). From what it looks like, you're just doing send/send/send/etc on your REQ socket without ever receiving a response. If you don't care about a response from your peer, then you must receive and discard it or use DEALER (or ROUTER, but DEALER makes more sense in your current diagram).

I've created a diagram of how I would accomplish this architecture below - using your basic process structure.

Broker T1         Broker T2                Broker T3                Broker T4
(PUB*)------>(*SUB)[--](DEALER*)   -->(*SUB)[--](DEALER*)   -->(*SUB)[--](DEALER*)
       |_____________________||____|                  ||    |                  ||
       |_____________________||_______________________||____|                  ||
                             ||                       ||                       ||
     ========================||     ==================||            ===========||=
   ||             ||              ||              ||              ||              ||
   ||             ||              ||              ||              ||              ||
   ||             ||              ||              ||              ||              ||
(REP*)         (REP*)          (REP*)          (REP*)          (REP*)          (REP*)
 S1P1           S1P2            S2P1            S2P2            S3P1            S3P2

So, the main difference is that I've ditched your (SUB copyLoop=> REQ) step. Whether you choose XPUB/XSUB vs PUB/SUB is up to you, but I would tend to start simpler unless you currently want to make use of the extra features of XPUB/XSUB.

Obviously this diagram doesn't deal with how information enters your broker, where you currently show an XSUB socket - that's out of scope for the information you've provided thus far, presumably you're able to receive information into your broker successfully already so I won't deal with that.

I assume your broker threads that are dedicated to each service are making intelligent choices on whether to send the message to their service or not? If so, then your choice of having them subscribed to everything should work fine, otherwise more intelligent subscription setups might be necessary.

If you're using a REP socket on your service processes, then the service process must take that message and deal with it asynchronously, never communicating back any details about that message to the broker. It must then respond to each message with an acknowledgement (like "RECEIVED") so that it follows the strict receive/send/receive/send pattern for REP sockets.

If you want any other type of communication about how the service handles that message sent back to the broker, REP is no longer the appropriate socket type for your service processes, and DEALER may no longer be the correct socket type for your broker. If you want some form of load balancing so that you send to the next open service process, you'll need to use ROUTER/REQ and have each service indicate its availability and have the broker hold on to the message until the next service process says its available by sending results back. If you want some other type of message handling, you'll have to indicate what that is so a suitable architecture can be proposed.

Jason
  • 13,606
  • 2
  • 29
  • 40
  • Thank you so much for the detailed answer, I clearly didn't understand the fact that REQ was expecting a reply before it could send another message. I've implemented the diagram and while there are no errors anymore, the messages don't reach their destination. The SUB/DEALER proxy is subscribed to everything. Can you confirm that the SUB/PUB and SUB/DEALER brokers are both running ZMQ.proxy? – Spiff Mar 11 '16 at 16:13
  • Using a ZMQ proxy is probably the preferred way to do it. Do you know where the message is getting stuck? How far does it make it through? – Jason Mar 11 '16 at 16:24
  • I have no idea, what is the best way to trace it? – Spiff Mar 11 '16 at 16:25
  • ZMQ.proxy() has a capture socket parameter, I'll try using this. – Spiff Mar 11 '16 at 16:51
  • After looking at your test program, it looks like you *may* be falling victim to [a slow joiner](http://zguide.zeromq.org/page:all#Getting-the-Message-Out) - your "event publisher" thread connects the publisher and then *immediately* sends out a single message - the subscriber may not be finished setting up its subscription by the time you attempt to send your first message. `PUB` sockets don't wait for the peer to finish connecting, they just drop the message if there are no subscribers. I recommend you wait a sec before sending your message, or sending a bunch of messages in a loop. – Jason Mar 14 '16 at 05:47
1

Clearly I got mixed up with a few elements:

  • Sockets have the same API whether you're using it as a client-side socket (Socket.connect) or a server-side socket (Socket.bind)
  • Sockets have the same API regardless of the type (e.g. Socket.subscribe should not be called on a PUSH socket)
  • Some socket types require a send/receive response loop (e.g. REQ/REP)
  • Some nuances in communication patterns (PUSH/PULL vs ROUTER/DEALER)
  • The difficulty (impossiblity?) in debugging a ZeroMQ setup

So a big thanks to Jason for his incredibly detailed answer (and awesome diagram!) that pointed me to the right direction.

I ended up with the following design:

  • broker thread 1 is running a fan-out XSUB/XPUB proxy on bind(localhost:6000) and bind(localhost:6001)
  • broker thread 2 is running a queuing SUB/PUSH proxy on connect(localhost:6001) and bind(localhost:6002); broker threads 3 and 4 use a similar design with different bind port numbers
  • message producers connect to the broker using a PUB socket on connect(localhost:6000)
  • message consumers connect to the broker queuing proxy using a PULL socket on connect(localhost:6002)

On top of this service-specific queuing mechanism, I was able to add a similar service-specific fan-out mechanism rather simply:

  • broker thread runs a SUB/PUB proxy on connect(localhost:6001) and bind(localhost:6003)
  • message producers still connect to the broker using a PUB socket on connect(localhost:6000)
  • message consumers connect to the broker fan-out proxy using a SUB socket on connect(localhost:6003)

This has been an interesting ride.

Spiff
  • 2,266
  • 23
  • 36