I am working on a Flask-SocketIO application that integrates with zmq. The basic premise of the app is that a zmq message is received by the Flask-SocketIO web server, then that zmq message is converted to a SocketIO message which is sent along to the client (browser). I have the app working but not exactly the way that I want it to be working.
There is a need for the zmq event listener to be on a separate process than the main server process. I was able to use both Redis and RabbitMQ as a message queue for facilitating SocketIO emits from non-server processes. OK, great. So what's the problem?
The problem is that I'd really like to use zmq as a message queue instead of Redis or RabbitMQ since I'm already integrating with zmq in my app. So I read in the Flask-SocketIO docs that Kombu is the mechanism for supporting other types of message queues. OK, cool. But then I notice that zmq as a transport has been removed from the latest release of Kombu. And as far as I can tell it was only experimental when it was a transport option, like in Kombu 3.0.37.
My first approach was to just try zmq as a message transport via Kombu 3.0.37, but that is not working. I'm still trying to determine exactly why that is. But my best guess right now after looking at the source code a bit is that multiple processes are trying to open a PULL socket on the same port, which just doesn't work, even in a simple independent example. And that makes sense. On that front, my next step is to manually create a PULL socket independently of the zmq transport code, and somehow pass that in as a type of singleton for the zmq transport code to use.
Another approach that I'm working is getting zmq to work as a transport in a Hello World Kombu example. I swapped out the connection string in the sample code with zmq+tcp://localhost. This is where it becomes clear that I'm not understanding how to use zmq as a message queue transport. If I run the publisher code, I can send a message. But when I run the client code a few seconds later, it says that the queue is empty. That makes me think that in order for zmq to work as a transport, I may need an external zmq message broker of some sort, which I'm assuming I would need to put together on my own. But I haven't yet got my head around how that would work.
Any suggestions? Is this a waste of time to try to pursue zmq as a transport, or should I continue down this path? Miguel Grinberg (author of Flask-SocketIO) has graciously provided some direction, but I wanted to branch out a little to see if anyone else had some thoughts on the matter.
