Python zeromq -- Multiple Publishers To a Single Subscriber?

Question

I'd like to write a python script (call it parent) that does the following:

(1) defines a multi-dimensional numpy array

(2) forks 10 different python scripts (call them children). Each of them must be able to read the contents of the numpy array from (1) at any single point in time (as long as they are alive).

(3) each of the child scripts will do it's own work (children DO NOT share any info with each other)

(4) at any point in time, the parent script must be able to accept messages from all of its children. These messages will be parsed by the parent and cause the numpy array from (1) to change.

---

How do I go about this, when working in python in a Linux environment? I thought of using zeroMQ and have the parent be a single subscriber while the children will all be publishers; does it make sense or is there a better way for this?

Also, how do I allow all the children to continuously read the contents of the numpy array that was defined by the parent ?

Answer 1

The sub channel doesn't have to be the one to bind, so you can have the subscriber bind, and each of the children pub channels can connect to that and send their messages. In this particular case, I think the multiprocessing module is a better fit, but I thought it bore mentioning:

import zmq
import threading

# So that you can copy-and-paste this into an interactive session, I'm
# using threading, but obviously that's not what you'd use

# I'm the subscriber that multiple clients are writing to
def parent():
    context = zmq.Context()
    socket = context.socket(zmq.SUB)
    socket.setsockopt(zmq.SUBSCRIBE, 'Child:')
    # Even though I'm the subscriber, I'm allowed to get this party 
    # started with `bind`
    socket.bind('tcp://127.0.0.1:5000')

    # I expect 50 messages
    for i in range(50):
        print 'Parent received: %s' % socket.recv()

# I'm a child publisher
def child(number):
    context = zmq.Context()
    socket = context.socket(zmq.PUB)
    # And even though I'm the publisher, I can do the connecting rather
    # than the binding
    socket.connect('tcp://127.0.0.1:5000')

    for data in range(5):
        socket.send('Child: %i %i' % (number, data))
    socket.close()

threads = [threading.Thread(target=parent)] + [threading.Thread(target=child, args=(i,)) for i in range(10)]
for thread in threads:
    thread.start()

for thread in threads:
    thread.join()

In particular, the Core Messaging Patterns part of the documentation discusses the fact that for the patterns, either side can bind (and the other connect).

Answer 2

I think it makes more sense to use PUSH/PULL sockets, as you have a standard Ventilator - Workers - Sink scenario, except that the Ventilator and the Sink are the same process.

Also, consider using the multiprocessing module instead of ZeroMQ. It will probably be a bit easier.

Answer 3

In ZeroMQ there can only be one publisher per port. The only (ugly) workaround is to start each child PUB socket on a different port and have the parent listen on all those ports.

but the pipeline pattern describe on 0MQ, user guide is a much better way to do this.