How can I both Send and Receive from a Router socket in ZeroMQ or NetMQ?

Question

I have a Dealer <--> Router setup in NetMQ v4 where I can asynchronously send and receive messages in any direction with no problems.

I now want to formalize that into an abstraction where the server (Router) listens for any incoming message but it also need to on demand broadcasts messages to any of the connected clients (Dealers).

I am trying to avoid using Pub <--> Sub sockets as I need the subscribers to also send messages to the server. The closest pattern to what I am trying to achieve is a WebSocket client-server communication.

The first part of listening to the client messages are done in something like:

using (var server = new RouterSocket("@tcp://*:80"))
{
    var addresses = new HashSet<string>();
    while (true)
    {
        var msg = server.ReceiveMultipartMessage();

        var address = Encoding.UTF8.GetString(msg[0].Buffer);
        var payload = Encoding.UTF8.GetString(msg[2].Buffer);
        Console.WriteLine("[Server] - Client: {0} Says: {1}", address, payload);

        var contains = addresses.Contains(address);
        if (!contains) { addresses.Add(address); }            

        msg.Clear();
        msg.Append(address);
        msg.AppendEmptyFrame();
        msg.Append("Reply for: " + address);
        server.SendMultipartMessage(msg);
    }
}

Now given that the sockets are not Thread-Safe, I am stuck on finding a way to broadcasts messages (coming from a different thread on demand) to all the clients.

I can probably use the TryReceiveMultipartMessage method in the loop instead with a set timeout after which I can check a queue for any broadcast messages and then loop through each client sending such message. Something like:

using (var server = new RouterSocket("@tcp://*:80"))
{
    var addresses = new HashSet<string>();

    var msg = new NetMQMessage();
    while (true)
    {
        var clientHasMsg = server.TryReceiveMultipartMessage(TimeSpan.FromSeconds(1), ref msg);
        if (!clientHasMsg)
        {
            // Check any incoming broacast then loop through all the clients
            // sending each the brodcast msg
            var broadMsg = new NetMQMessage();
            foreach (var item in addresses)
            {
                broadMsg.Append(item);
                broadMsg.AppendEmptyFrame();
                broadMsg.Append("This is a broadcast");
                server.SendMultipartMessage(broadMsg);
                broadMsg.Clear();
            }

            // Go back into the loop waiting for client messages
            continue;
        }

        var address = Encoding.UTF8.GetString(msg[0].Buffer);
        var payload = Encoding.UTF8.GetString(msg[2].Buffer);
        Console.WriteLine("[Server] - Client: {0} Says: {1}", address, payload);

        var contains = addresses.Contains(address);
        if (!contains) { addresses.Add(address); }

        msg.Clear();
        msg.Append(address);
        msg.AppendEmptyFrame();
        msg.Append("Reply for: " + address);
        server.SendMultipartMessage(msg);
    }
}

This somehow does not feel right mainly due to:

What value for the timeout is a good value? 1 sec, 100 ms etc;
Is this the most efficient/performing solution as this program will be used to have 100k+ clients connected with each sending thousands of messages per second.

Any pointers on what's the best approach to this is very much appreciated.

score 2 · Accepted Answer · answered Oct 11 '16 at 19:58

2

You can use netmqqueue, it multi producer single consumer queue. You can add it to NetMQPoller and enqueue from multiple threads without lock.

answered Oct 11 '16 at 19:58

somdoron

4,653
2
16
24

I was just reading your blog on `Device` and thought it would be a good choice until you mentioned the `Queue` :-) Have you got any examples somewhere (other than the few lines on http://netmq.readthedocs.io)? – MaYaN Oct 11 '16 at 20:33
Okay, I think I can get this working with no additional example just one more question, what's the difference between NetMQQueue and the NetMQSscheduler? Is the scheduler obsolete in v4? – MaYaN Oct 11 '16 at 20:55
NetMQScheduler is obsolete (now part of NetMQPoller), anyway NetMQScheduler is queue of tasks, NetMQQueue is queue of any type. – somdoron Oct 12 '16 at 15:30
The linked example shows a single dequeue per receive ready, but socket notifications (at least) are edge triggered. Is this a problem? – SensorSmith Aug 08 '18 at 17:18
The OPs Q is directly on point to minimal real world usage of 0MQ, but I can find no example of using a queue or second socket to pass outgoing messages to the single thread allowed to touch the original socket AND any blocking read (or hitting a send HWM) means you cannot shutdown because you cannot close the socket from another thread (or if you do it does not unblock). So... can you provide a fuller example using NetMQPoller, a NetMQQueue, some "purposeful" socket, and a way to shut it all down? – SensorSmith Aug 08 '18 at 17:22
The NetMQQeue is not edge triggered. – somdoron Oct 11 '18 at 10:14
Sorry, I cannot find an example at the moment. You can check the unit tests of NetMQQueue: https://github.com/zeromq/netmq/blob/e4234fc80527e17461673beba18d686aa31f0172/src/NetMQ.Tests/NetMQQueueTests.cs – somdoron Oct 11 '18 at 10:21

score 0 · Answer 2 · answered Oct 11 '16 at 16:31

0

I think PUB/SUB is a proper approach for your requirements of 100k+ clients. Nevertheless, it doesn't mean that you can't communicate back to server: use DEALER/ROUTER. Why do you think this solution is not acceptable?

answered Oct 11 '16 at 16:31

iiwaasnet

51
1
8

I am not sure I understand correctly. Are you saying that doing Pub/Sub using Dealer/Router based on my above solution looks reasonable to you? – MaYaN Oct 11 '16 at 17:54

How can I both Send and Receive from a Router socket in ZeroMQ or NetMQ?

2 Answers2