3

Our software implement a actor model system, and we allocate/deallocate the small object very often.I am very sure the each object be destroyed without memory leak. ( I have used valgrind and tcmalloc tool to check the memory leak in my software. No leak be found.)

When we change to use tcmalloc for replacing the malloc in glibc, we found the memory continually increasing until the process be killed by OOM(Out of Memory). Then we found glibc also have same issue, but increase rate is less than tcmalloc.

I used the malloc_stats() to show the memory information

after first execution(top show 0.96G)'

  • MALLOC: 960110592 ( 915.6 MB) Heap size
  • MALLOC: 15886016 ( 15.2 MB) Bytes in use by application
  • MALLOC: 907419648 ( 865.4 MB) Bytes free in page heap
  • MALLOC: 0 ( 0.0 MB) Bytes unmapped in page heap
  • MALLOC: 27121208 ( 25.9 MB) Bytes free in central cache
  • MALLOC: 151040 ( 0.1 MB) Bytes free in transfer cache
  • MALLOC: 9532680 ( 9.1 MB) Bytes free in thread caches
  • MALLOC: 14275 Spans in use
  • MALLOC: 27 Thread heaps in use
  • MALLOC: 7602176 ( 7.2 MB) Metadata allocated

After 5th same execution(top show 1.2G)

  • MALLOC: 1173131264 ( 1118.8 MB) Heap size
  • MALLOC: 18001048 ( 17.2 MB) Bytes in use by application
  • MALLOC: 1082458112 ( 1032.3 MB) Bytes free in page heap
  • MALLOC: 21168128 ( 20.2 MB) Bytes unmapped in page heap
  • MALLOC: 37992328 ( 36.2 MB) Bytes free in central cache
  • MALLOC: 252928 ( 0.2 MB) Bytes free in transfer cache
  • MALLOC: 13258720 ( 12.6 MB) Bytes free in thread caches
  • MALLOC: 17651 Spans in use
  • MALLOC: 27 Thread heaps in use
  • MALLOC: 8126464 ( 7.8 MB) Metadata allocated

We can seen from thus data. after 5th same behavior, only 17.2 be used in our software. But the tcmalloc hold 1.1G memory without return to system. Of course, It does not matter tcmalloc hold those memory. But it keep increasing when our program be killed by OOM(actual used memory is less than 1G).

We have doubt that it is related to heap fragmentation. Anybody have some experience could share with us? I think I have same situation as https://bugzilla.redhat.com/show_bug.cgi?id=843478

Thanks very much.

xiaoningyb
  • 41
  • 1
  • 3
  • Why are you so sure there is absolutely no memory leaks??? – Basile Starynkevitch Mar 22 '13 at 09:11
  • 1
    "I am very sure the each object be destroyed without memory leak." Im very sure that there is a greater chance of bug in your code than in the tcmalloc. :) Like A says try valgrind, it is quite easy to run actually – NoSenseEtAl Mar 22 '13 at 09:12
  • BTW, what software are you developing? Is it free software? I am interested in actor systems implemented in C or C++.... – Basile Starynkevitch Mar 22 '13 at 09:19
  • @NoSenseEtAl Yes, I have used valgrind and tcmalloc tool to check the memory leak in my software. No leak be found. – xiaoningyb Mar 22 '13 at 09:49
  • @NoSenseEtAl Also, after each execution we may added some cache for store something. Bug according to "18001048 ( 17.2 MB) Bytes in use by application" which reported by malloc_stats(), i am sure we could not hold such a large number of memory to make OS kill this process by OOM. – xiaoningyb Mar 22 '13 at 09:51
  • this is higely unlikely but maybe your usage pattern really messes up with tcmalloc algs and is has a huge overhead per each byte you actually allocate... again doesnt feel really realistic. – NoSenseEtAl Mar 22 '13 at 10:38
  • btw there is one thing about tcmalloc that puzzled me a long time ago: "TCMalloc currently does not return any memory to the system. " but it should still reuse freed memory... – NoSenseEtAl Mar 22 '13 at 11:10
  • @NoSenseEtAl: Perhaps `tcmalloc` never calls `munmap` – Basile Starynkevitch Mar 22 '13 at 11:57
  • btw one more leet technique... use CRTP to count alive instances of your objects if you have low amound of diff objects – NoSenseEtAl Mar 22 '13 at 12:32

2 Answers2

1

I would suggest using Boehm's conservative GC and use GC_MALLOC and GC_MALLOC_ATOMIC instead of malloc in your application (and GC_FREE instead of free, but you could then even avoid any explicit free-ing, the GC would do them). Or maybe use valgrind (with the system Glibc malloc) to find memory leaks. If using Boehm's GC, don't forget to explicitly clear the allocated memory zones.

It would be preferable to make sure that your many small objects have coarse grained sizes. E.g. allocate objects of 8 or 12 or 16 words, not objects of 8, 9, 10, 11, 12, 13, 14, 15 or 16 words... For instance, you might allocate only zones of size some power of 2 or 3 times a power of 2.

Also, don't forget you can limit the memory space with setrlimit(2), e.g. with the ulimit builtin of the bash running in your terminal. This should ease testing. Also, perhaps using pmap or /proc/$(pidof yourapp)/maps could help you to understand the address space used.

PS. Neither Boehm GC, nore any kind of malloc (including tcmalloc or Glibc malloc) can tackle with memory fragmentation. If you suspect fragmentation you have to move memory zones in the address space (i.e. you may want to code your own precise, copying, generational GC or re-use some existing one).

Basile Starynkevitch
  • 223,805
  • 18
  • 296
  • 547
  • Thanks for your idea. I have used valgrind and tcmalloc tool to check the memory leak in my software. No leak be found. I think GC and fixed size object may resolve some question. But we already have manager the object in our software using reference_count in C++. So it is very hard to do thus big change. – xiaoningyb Mar 22 '13 at 09:52
  • Then you might indeed be hurt by memory fragmentation, and you cannot do much easily (because you need to move memory zones). If you have some persistence mechanism, you could persist your state on file, stop the program, and restart it with the persisted state on disk. – Basile Starynkevitch Mar 22 '13 at 10:03
1

tcmalloc tries to do some smart things to anticipate your memory use, but it isn't very good about releasing memory back to the system even after you have freed it. in fact, it might be resident in memory and leading to OOM.

do this :

MallocExtension::instance()->ReleaseFreeMemory();

when your application has expectedly freed memory that it expects to not use soon.

look at the section "Releasing memory back to the system" for more information http://google-perftools.googlecode.com/svn/trunk/doc/tcmalloc.html

there are other methods, such as setting the Release rate, but as a start you can sprinkle this in your code and inspect if resident memory goes down as you expect.

Michael Xu
  • 557
  • 1
  • 5
  • 14