1

I have a process that consumes a lot of memory on startup, but frees most of that memory after the process is bootstrapped. I see the following in the TCMalloc stats printed afterwards:

#012MALLOC:       16635888 (   15.9 MiB) Bytes in use by application
#012MALLOC: +            0 (    0.0 MiB) Bytes in page heap freelist
#012MALLOC: +     20007352 (   19.1 MiB) Bytes in central cache freelist
#012MALLOC: +     26367680 (   25.1 MiB) Bytes in transfer cache freelist
#012MALLOC: +      7030680 (    6.7 MiB) Bytes in thread cache freelists
#012MALLOC: +      1978560 (    1.9 MiB) Bytes in malloc metadata
#012MALLOC: =     72020160 (   68.7 MiB) Actual memory used (physical + swap)
#012MALLOC: +    239607808 (  228.5 MiB) Bytes released to OS (aka unmapped). ***
#012MALLOC: =    311627968 (  297.2 MiB) Virtual address space used

Here we see that there's ~228 MB "released to the OS", but then it also shows that this is still part of the process' virtual address space. This is corroborated by the fact that the VSZ stat seen with ps aux remains high after this logging is seen.

When I run the same program with strace, I can see that: all of the calls to brk are increasing, and that the calls to munmap (in terms of sizes) do not add up to the amounts mmap-ed (and showed above). What's worse is that VSZ increases slowly over time as my process does background work, and the logging shows that (***) increases correspondingly when these stats are logged.

So my questions are: What does this value represent? Has my memory been freed or not? What can I do to prevent this consumption from growing?

Nizbel99
  • 123
  • 6

1 Answers1

1

What does this value represent?

It represents the amount of memory that TCMalloc has told the system that it does not need and that the system may use for other purposes.

Has my memory been freed or not?

No. The OS has decided that making it free just to have to make it used again when the program needs it is a waste of effort and has decided instead to switch it directly from one use to another use in a single step rather than going through twice the effort of making it free just to have to make it un-free to use it.

What can I do to prevent this consumption from growing?

Why would you want to? It just makes it easier if the program needs the memory later, minimizes contention on the system's free list, and has no harmful effects at all. TCMalloc has told the OS (via madvise(DONTNEED)) that the OS may recover the memory and the OS has made the decision that it's not a good idea to make it free just to have to make it used again when it's needed. Do you have some good reason to think the OS is wrong?

It's much easier to just directly transition memory from one us to another in a single step than go through the two steps of making it free just to have to make it used again. The free list can get contended under load and it's much simpler not to use it.

You can force the OS to free it by running some program that consumes a lot of memory and then terminates. That will force the OS to transition the memory to that process and then free it when that process terminates. But this would provide no benefit at all and would be a lot of effort just to eventually increase contention in the memory manager. There is no issue here.

David Schwartz
  • 179,497
  • 17
  • 214
  • 278
  • But why is this memory not reused? Over time, I can observe the value "Bytes released to OS (aka unmapped)" grow. Why does the process get new memory from the OS instead of reusing what it previously told the OS it no longer needed? – Nizbel99 Jun 03 '21 at 22:48
  • 1
    @Nizbel99 The OS tries as hard as it can to make as much memory be in use as possible. Free memory has no effect on performance and takes more steps to put in use than memory that is already in use. The OS is taking the most efficient path. If the process later needs more memory, the more the OS has already assigned it, the better. In pretty much every other scenario, it's either a tie or still better to not have to make the memory free and then unfree again. What benefit do you imagine will come from freeing the memory? – David Schwartz Jun 03 '21 at 22:53
  • Thank you for the explanation. Since the memory is not truly getting freed, does this mean it is a memory leak? – Nizbel99 Jun 07 '21 at 17:24
  • 1
    @Nizbel99 It depends what you mean by "the memory". If you mean the physical memory used, then no, the OS is still correctly tracking it and can reuse it. If you mean the process address space used, then again no, the process is still correctly tracking it and can reuse it. This is everything working perfectly as designed. Process memory pressure will cause the address space to be reused. System memory pressure will cause the RAM to be reused. This is peak efficiency, not doing unnecessary work that may just need to be reversed later. – David Schwartz Jun 07 '21 at 17:32