Router/Dealer proxy not delivering messages

Question

I'm working on distributed application. The networking is based on ZMQ (jeromq), here's my architecture:

N Clients (Dealer socket) <---> (Router) Proxy (Dealer) <---> 1 (Dealer) Reciving Worker 
                                                        <---> N (Dealer) Processing Workers

• So my Receiving Worker is running in a separate thread listening for messages and putting them into the queue.
• The Processing Workers pick the message from the queue, do some processing and send it back to the Client (processing worker is connected to the proxy backend)

When a client sends a message, it is received by the receiving worker, placed in the queue, the processing worker picks it up does some processing and sends it back, the client receives the response after which it sends another request, that request is not being received by the receiving worker but it is sent by the client. Basically only the first 'round' works. What am i missing here ?!

Answer 1

What you're missing is that Dealer sockets send out messages to any connected client in a round robin fashion. The upshot is that your architecture won't work the way you intend.

To start, I'll write up a more accurate diagram of your intended architecture (for the moment, I'm ignoring socket types):

Client <-----> Broker ------> Receiver
                  ^              |
                  |              |
                  \              v
                   ---------- Processor

When you send the first message, it gets sent to the receiver - because it's the first connected socket in the queue. When you send the second message, the DEALER socket on your broker is sending to the next socket in the queue, in a round-robin fashion, which is the first processor. Basically, the problem is that your broker isn't able to distinguish between your "receiver" socket and your "processor" sockets, it treats them all as the same thing.

There are a couple ways you could handle this. First of all, you probably don't really need your "receiver" socket as a go-between, you may be able to communicate directly back and forth between your broker and your processing workers. I'd suggest that first, as that's typically how these types of architectures work.

You could also add another socket to your broker, have one socket to send to the receiver and one to receive from the processors.

Or, you could replace your DEALER socket on your broker with another ROUTER socket, which will allow you to directly address the receiver socket every time.

---

Follow up, to address the question in your comment:

Each of your workers having a different responsibility is a good reason to replace the DEALER socket on your broker with a ROUTER socket.

My off-the-shelf recommendation is to rearchitect around the Majordomo Protocol (later version can be found here, and talked about in the ZMQ guide here, with Java code samples). It does exactly what you seem to be looking for, and does it well.

But, at a simpler level, what you'll be doing is the following:

1. Spin up your broker, with ROUTER sockets on both the front and back ends - both of these sockets are bind()-ed.
2. Spin up your workers. Each worker is aware of its own responsibility. Remove the "receiver", we won't be using it.
3. connect() your workers do your broker.
4. Each worker sends a message to the broker, telling the broker what that worker is responsible for. The broker keeps a record of each worker by ID, and what job that worker is responsible for.
5. When a broker wants to send a specific job, it looks up which worker is responsible for that job, and sends the job to it.
6. If the worker is available immediately, it works on it, otherwise it holds onto the job until it's ready to process it.
7. When the worker is finished, it sends the job back and continues working on new jobs if it has any.

All of that is what the Majordomo Protocol implements, complete with other necessary elements like heartbeating that make a fully fleshed out ZMQ application. That's why I recommend you go straight to that protocol, if it serves the needs you're trying to accomplish, rather than trying to build your own.