2

I want to investigate the memory usage of a python program that uses numpy.memmap to access data from large files. Is there a way to check the size in memory that a memmap is currently using? I tried sys.getsizeof on the numpy object and the _mmap attribute of the object, but both gave the same very small size regardless of how much the memmap object had been used.

Colin
  • 10,447
  • 11
  • 46
  • 54
  • Did you ever find a nice solution? I now have the same problem that `sys.getsizeof` returns a much too small value for `numpy.memmap` instances. – hans_meine Aug 25 '22 at 15:37

1 Answers1

0

I found that array.base gives you a mmap.mmap instance (assuming that array is a numpy.memmap instance). mmap.mmap supports len, so I am now using:

def recursive_size_of(list_or_array) -> int:
    result = sys.getsizeof(list_or_array)
    if isinstance(list_or_array, list):
        result += sum(map(recursive_size_of, list_or_array))
    elif isinstance(list_or_array, mmap.mmap):
        result += len(list_or_array)
    else:
        base_array = getattr(list_or_array, 'base', None)
        if base_array is not None:
            result += recursive_size_of(base_array)
    return result

If you know that your array is exactly a numpy.memmap and feel confident about making that assumption in your code, len(array.base) would suffice.

hans_meine
  • 1,881
  • 1
  • 17
  • 29