ipython parallel and non-copying sends of numpy arrays

Question

I'm trying to understand what happens with IPython parallel's non-copying send/receive of numpy arrays. I understand that the non-copying receive of a message is read-only, but this leads me to expect that the numpy array that I receive acts like a view, pointing to the original array (at least on shared memory machines). I would then expect that if the numpy array is modified on one of the compute nodes, my view of that array in the notebook would be updated, however, that doesn't seem to be the case. To demonstrate what's confusing me, the code

from IPython.parallel import Client
dv = Client()[:]

with dv.sync_imports():
    import numpy 

dv.execute('a = numpy.zeros(2)')
print dv['a']
mya = dv['a'][1] # my copy of the array in the notebook
# mya[0] = -1 # can't assign to mya because it is read-only
dv.execute('a[1] = 1') # update the arrays on the compute nodes
print mya # value of mya is not updated
print dv['a'] # even though the arrays are updated on the compute nodes

has output

importing numpy on engine(s)
[array([ 0.,  0.]), array([ 0.,  0.]), array([ 0.,  0.]), array([ 0.,  0.])]
[ 0.  0.]
[array([ 0.,  1.]), array([ 0.,  1.]), array([ 0.,  1.]), array([ 0.,  1.])]

I would think that the point of having the non-copying sends is to save memory by having every view of the array point to the same chunk of memory, but that doesn't seem to be the case here. What is happening under the hood, and how am I misunderstanding the use of the non-copying sends?

The context of my main question above is a shared memory environment (my laptop), but a possibly related question I have is how do non-copying sends of numpy arrays work and what is their use in the distributed memory case, where compute nodes are different physical machines and you must (I assume) send the numpy arrays over the network (effectively making a copy on the receiving machine), so having a view to the original memory location wouldn't make sense.

Answer 1

A non-copying send in the context of IPython.parallel (or the underlying zeromq) does not mean that memory is shared with the destination. IPython.parallel is not based on a shared memory system but instead on explicit message passing.

It instead means that the network interface directly retrieves its data from the array you are providing instead of making a copy of the data into its own buffers in the network stack. The transfer to the destination process or thread will still always be a copy, even if it is on the same machine.

So the receiver of the data can modify the data without changing the data on the senders side. But the sender has to take care to not modify the buffer it gave to the network interface before that is done copying the data to the destination. With pyzmq this can be done by with the track keyword argument which gives you an object that you can poll for the transfer to be completed, and the buffer again being available for modification.

Non-copying sends are a dangerous optimization as you have to be very careful about which buffers are available for writing and which aren't. You should only use them when you have clearly pinpointed the additional copy to the network stack to be a performance problem.