2

I have a numpy structured array whose dtype has mutiple fields with different data type. And I want to update this structured numpy array in my python multiprocesses. And I think I am going to use Pool. What way I can use to share this numpy structured array?
I have read some pages about wrapping numpy array in Ctype Array, but I don't think there is suitable ctype for this.
Can I pass this array as pointer? I also want to write the function as worker process in cython.

Code sample as below:

import numpy as np
import multiprocessing as mp

def modi(arg):
    i,pt = arg
    if pt['mdistance'][0] == 4:
        return i
    pt['distance'] += 2

if __name__ == '__main__':
    eetype=[('coordinate', 'f8', (2,)), ('file_id', '<U11'), ('distance', 'f8', (2,))]
    aa= np.zeros(5, dtype=eetype)
    aa['file_id']=np.array(['aa', 'bb', 'cc', 'dd', 'ee'])
    aa['coordinate']=np.array([[1,1],[2,2],[3,3],[4,4],[5,5]])
    aa['distance']=np.array([[1,1],[2,2],[3,3],[4,4],[5,5]])
    print(aa)
    with mp.Pool(4) as p:
        a=list(p.map(modi, enumerate(aa)))
        #this aa array should be shared in different processes
Cœur
  • 37,241
  • 25
  • 195
  • 267
lX-Xl
  • 160
  • 1
  • 6
  • A couple of recent SO questions wanted to use `mp.Array` to share numpy arrays. That doesn't work as hoped because that class creates a new buffer in memory, and copies an iterable to it. So it doesn't preserve `ndarray` attributes. And it isn't a `view` or pointer. – hpaulj May 12 '18 at 17:06
  • Processes do not share memory. Read up on multithreading – danny May 14 '18 at 08:58
  • @danny processes can share memory, e.g. using `mp.Array`. The question is how to do it with a structured array (which is exactly what I'm trying to figure out too) – Just Me May 23 '23 at 07:34

0 Answers0