By reading documentation to IPython Parallel it says that the buffers are copied between cores. However, it says that by using track = True one can edit the buffers in place.
The following example doesn't work for me, even with track = True:
import numpy as np
from IPython.parallel import Client
A = np.zeros((10,10), int)
rc = Client()
dview = rc[:]
def ptest(a):
a[0] = 5
dview.track = True
r = dview.map_sync(ptest, A)
r.wait_for_send()
It keeps prompting the error that a is not writable (assigment destination is read-only).
The problem I'm facing is that I have very large 3D matrices and I would like to modify them in place. My idea was to use a map-like approach to send individual 2D slices to the clusters and modify them in-place.
Is there a way of achieving this with IPython Parallel? Or is it virtually impossible to send writeable buffers and the only way is through gathering the results from copied buffers?
If this is not a possibility, is there a way with IPython Notebook to asynchronously get the results from cores and update them with a callback function? This is the pseudocode of something that would be OK. Is there anything like this available in IPython Parallel?:
def edit(a):
a[0] = 5
return a
def callback(i, a):
A[i] = a
r = dview.map_async(edit, A, callback)
r.wait()
The idea of the above code is to have a function that returns modified rows from a matrix in parallel with a callback that asynchronously updates the rows of an output buffer (in the current machine).
In other words, something like this with IPython Parallel instead of multiprocessing would work.
