1

I am trying to process some arrays with Pool amd many workers. As I am working with relatively big arrays I use shared memory which is converted to numpy arrays later in the called function:

shared_arr = Array(ctypes.c_ulong, size[0] * size[1])
img = tonumpyarray(shared_arr, size)

shared_arr_l = Array(ctypes.c_ulong, size[0] * size[1])
labimg = tonumpyarray(shared_arr_l, size)

The Pool function is:

with closing(Pool(initializer=init, initargs=(shared_arr, shared_arr_l), processes=n_cores)) as p:
        p.starmap_async(classific, [(p, size) for p in props1])
p.join()

It works for small data but for large data I get an AssertionError for every worker:

Process SpawnPoolWorker-1:
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\Lib\multiprocessing\process.py", line 258, in _bootstrap
    self.run()
  File "C:\ProgramData\Anaconda3\Lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "C:\ProgramData\Anaconda3\Lib\multiprocessing\pool.py", line 108, in worker
    task = get()
  File "C:\ProgramData\Anaconda3\Lib\multiprocessing\queues.py", line 335, in get
    res = self._reader.recv_bytes()
  File "C:\ProgramData\Anaconda3\Lib\multiprocessing\connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "C:\ProgramData\Anaconda3\Lib\multiprocessing\connection.py", line 318, in _recv_bytes
    return self._get_more_data(ov, maxsize)
  File "C:\ProgramData\Anaconda3\Lib\multiprocessing\connection.py", line 337, in _get_more_data
    assert left > 0
AssertionError

Is it a memory problem? RAM shouldn't be a problem - the computer has 128 GB while the two arrays are less than 2 GB and are stored in a shared memory. I tried to solve it but I am lost. Any help would be greatly appreciated.

marsmoon
  • 11
  • 3
  • possible duplication of [this](https://stackoverflow.com/questions/47692566/python-multiprocessing-apply-async-assert-left-0-assertionerror). please read through – Aneesh Damodaran Jul 12 '18 at 13:39

0 Answers0