Why does concurrent.futures increase vms memory?

Question

I find that when I use Python's concurrent.futures.ThreadPoolExecutor the vms memory use (as reported by psutil) increases dramatically.

In [1]: import psutil

In [2]: psutil.Process().memory_info().vms / 1e6
Out[2]: 360.636416

In [3]: from concurrent.futures import ThreadPoolExecutor

In [4]: e = ThreadPoolExecutor(20)

In [5]: psutil.Process().memory_info().vms / 1e6
Out[5]: 363.15136

In [6]: futures = e.map(lambda x: x + 1, range(100))

In [7]: psutil.Process().memory_info().vms / 1e6
Out[7]: 1873.580032

In [8]: e.shutdown()

In [9]: psutil.Process().memory_info().vms / 1e6
Out[9]: 1722.51136

This seems to be somewhat proportional to the number of threads.

Note that other memory attributes, like `rss`, return more modest values — MRocklin, Aug 14 '17 at 23:47

score 1 · Answer 1 · answered Aug 15 '17 at 01:49

You might be running into this (assuming you are on Linux):

https://siddhesh.in/posts/malloc-per-thread-arenas-in-glibc.html

This can bloat the virtual memory size even when RSS is not increasing much.

(Incidentally, VMS can be misleading in other situations, such as with CUDA, where the driver expands the virtual memory space of the process in order to create a unified address space with all of the CUDA devices in the system.)

Why does concurrent.futures increase vms memory?

1 Answers1