I am trying to process some arrays with Pool amd many workers. As I am working with relatively big arrays I use shared memory which is converted to numpy arrays later in the called function:
shared_arr = Array(ctypes.c_ulong, size[0] * size[1])
img = tonumpyarray(shared_arr, size)
shared_arr_l = Array(ctypes.c_ulong, size[0] * size[1])
labimg = tonumpyarray(shared_arr_l, size)
The Pool function is:
with closing(Pool(initializer=init, initargs=(shared_arr, shared_arr_l), processes=n_cores)) as p:
p.starmap_async(classific, [(p, size) for p in props1])
p.join()
It works for small data but for large data I get an AssertionError for every worker:
Process SpawnPoolWorker-1:
Traceback (most recent call last):
File "C:\ProgramData\Anaconda3\Lib\multiprocessing\process.py", line 258, in _bootstrap
self.run()
File "C:\ProgramData\Anaconda3\Lib\multiprocessing\process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "C:\ProgramData\Anaconda3\Lib\multiprocessing\pool.py", line 108, in worker
task = get()
File "C:\ProgramData\Anaconda3\Lib\multiprocessing\queues.py", line 335, in get
res = self._reader.recv_bytes()
File "C:\ProgramData\Anaconda3\Lib\multiprocessing\connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "C:\ProgramData\Anaconda3\Lib\multiprocessing\connection.py", line 318, in _recv_bytes
return self._get_more_data(ov, maxsize)
File "C:\ProgramData\Anaconda3\Lib\multiprocessing\connection.py", line 337, in _get_more_data
assert left > 0
AssertionError
Is it a memory problem? RAM shouldn't be a problem - the computer has 128 GB while the two arrays are less than 2 GB and are stored in a shared memory. I tried to solve it but I am lost. Any help would be greatly appreciated.