Task progress from a ZeroMQ worker

Question

Fairly new to ZeroMQ. I have a simple REQ/REP queue like below. I am using PHP but that doesn't matter as any language binding would be fine for me. This is client to request a task

$ctx = new ZMQContext();
$req = new ZMQSocket($ctx, ZMQ::SOCKET_REQ);
$req->connect('tcp://localhost:5454');
$req->send("Export Data as Zip");
echo $i . ":" . $req->recv().PHP_EOL;

And this is a worker to actually perform the task.

$ctx = new ZMQContext();
$srvr = new ZMQSocket($ctx, ZMQ::SOCKET_REP);
$srvr->bind("tcp://*:5454");
echo "Server is started at port $port" . PHP_EOL;
while(true)
{
    $msg = $srvr->recv();
    echo "Message = " . $msg . PHP_EOL;
    // Do the work here, takes 10 min, knows the count of lines added and remaining
    $srvr->send($msg . " is exported as zip file" . date('H:i:s'));
}

As the task of exporting data takes about 10 min, I want to connect to the server from a different client and get the progress/ percentage of the task done. I am wondering if that's even a valid approach.

I tried this approach where REQ/REP part works but I get nothing in PUB/SUB part

Server part

$ctx = new ZMQContext();
$srvr = new ZMQSocket($ctx, ZMQ::SOCKET_REP);
$srvr->bind("tcp://*:5454");

// add PUB socket to publish progress
$c = new ZMQContext();
$p = new ZMQSocket($c, ZMQ::SOCKET_PUB);
$p->bind("tcp://*:5460");

echo "Server is started at port 5454" . PHP_EOL;
$prog = 0;
while(true)
{
    $p->send($prog++ . '%'); // this part doesn't get to the progress client
    $msg = $srvr->recv();
    echo "Message = " . $msg . PHP_EOL;
    sleep(2);// some long task
    $srvr->send($msg . " Done zipping " . date('H:i:s'));
}

Progress client

$ctx = new ZMQContext();
$stat = new ZMQSocket($ctx, ZMQ::SOCKET_SUB);
$stat->connect('tcp://localhost:5460');
while (true){
    echo $stat->recv() . PHP_EOL; //nothing shows here
}

Request client

$ctx = new ZMQContext();
$req = new ZMQSocket($ctx, ZMQ::SOCKET_REQ);
$req->connect('tcp://localhost:5454');
for($i=0;$i<100;$i++){
    $req->send("$i : Zip the file please");
    echo $i . ":" . $req->recv().PHP_EOL; //works and get the output
}

Answer 1

The concept is feasible, some tuning needed:

All PUB counterparties have to setup any non-default subscription, via, at least an empty subscription .setsockopt( ZMQ_SUBSCRIBE, "" ) meaning receive all TOPICs ( none "filter"-ed out ).

Next, both PUB-side and SUB sides ought get .setsockopt( ZMQ_CONFLATE, 1 ) configured, as there is of no value to populate and feed all intermediate values into the en-queue/de-queue pipeline, once the only value is in the "last", most recent message.

Always, the non-blocking mode of the ZeroMQ calls ought be preferred ( .recv( ..., flags = ZMQ_NOBLOCK ) et al ) or the Poller.poll() pre-tests ought be used to sniff first for a (non)-presence of a message, before spending more efforts on reading its context "from" ZeroMQ context-manager. Simply put, there are not many cases, where blocking-mode service calls may serve well in a production-grade system.

Also some further tweaking may help the PUB side, in case a more massive "attack" comes from the un-restricted pool of SUB-side entities and PUB has to create / manage / maintain resources for each of these ( unrestricted ) counterparties.

Answer 2

You need only use PUB/SUB if there is more than one client wanting to receive the same progress updates. Just use PUSH/PULL for a simple, point to point transfer that works over tcp.

Philosophical Discussion

With problems such as this to solve there's two approaches.

1. Use additional sockets to convey additional message types,
2. Use just two sockets, but convey more than one message type through them

You're talking about doing 1). It might be worth contemplating 2), though I must emphasise that I know next to nothing of PHP and so don't know if there are language features that encourage one to have separate request and progress clients.

If you do, your original client needs a loop (after it has sent the request) to receive multiple messages, either progress update messages or the final result. Your server, whilst it is doing its 10 minute lookup, will send regular progress update messages, and the final result message at the end. You would probably use PUSH/PULL client to server, and the same again for the progress / result from the server back to the client.

It is architecturally more flexible to follow 2). Once you have a means of sending two or more message types through a single socket and of decoding them at the receiving end, you can send more. For example, you could decide to add a 'cancel' message from the client to the server, or a partial results message from the server back to the client. This is much easier to extend than to keep adding more sockets to your architecture simply because you want to add another message flow between the client and server. Again, I don't know enough about PHP to say that this would definitely be the right way of doing it in that language. It certainly makes a lot of sense in C, C++.

I find things like Google Protocol Buffers (I prefer ASN.1) very useful for this kind of thing. These allow you to define the types of messages you want to send, and (at least with GPB), combine them together inside a single 'oneof' (in ASN.1 one uses tagging to tell different messages apart). GPB and ASN.1 are handy because then you can use different languages, OSes and platforms in your system without really having to worry about what it is being sent. And being binary (not text) they're more efficient across network connections.