4

I run my C program on Suse Linux Enterprise that compresses several thousand large files (between 10MB and 100MB in size), and the program gets slower and slower as the program runs (it's running multi-threaded with 32 threads on a Intel Sandy Bridge board). When the program completes, and it's run again, it's still very slow.

When I watch the program running, I see that the memory is being depleted while the program runs, which you would think is just a classic memory leak problem. But, with a normal malloc()/free() mismatch, I would expect all the memory to return when the program terminates. But, most of the memory doesn't get reclaimed when the program completes. The free or top command shows Mem: 63996M total, 63724M used, 272M free when the program is slowed down to a halt, but, after the termination, the free memory only grows back to about 3660M. When the program is rerun, the free memory is quickly used up.

The top program only shows that the program, while running, is using at most 4% or so of the memory.

I thought that it might be a memory fragmentation problem, but, I built a small test program that simulates all the memory allocation activity in the program (many randomized aspects were built in - size/quantity), and it always returns all the memory upon completion. So, I don't think that's it.

Questions:

  1. Can there be a malloc()/free() mismatch that will lose memory permanently, i.e. even after the process completes?

  2. What other things in a C program (not C++) can cause permanent memory loss, i.e. after the program completes, and even the terminal window closes? Only a reboot brings the memory back. I've read other posts about files not being closed causing problems, but, I don't think I have that problem.

  3. Is it valid to be looking at top and free for the memory statistics, i.e. do they accurately describe the memory situation? They do seem to correspond to the slowness of the program.

  4. If the program only shows a 4% memory usage, will something like valgrind find this problem?

Alex Brown
  • 41,819
  • 10
  • 94
  • 108
Keith Hogan
  • 117
  • 1
  • 9
  • What happens if you wait an hour or so? Does the memory return? Does your program spawn other processes? How much ram do you have in your system? Are you using mmap? What happens if you remove the writes from your code? – Alex Brown Sep 08 '12 at 03:09
  • The code is too huge to post. No other processes spawned, just threads. 16GB RAM. Not using mmap. You mean remove the file writes? – Keith Hogan Sep 08 '12 at 03:26
  • 1
    Idle speculation, but when the program has exited, before you run it again, is the hard disk active? If so, does it stay active for ever or eventually stop, and if it eventually stops is your program still slow after it has stopped? I wonder whether 32 threads are producing rather more data than your disk can write, and so your program rapidly fills up RAM with 64GB of data that has been written out of the process but not yet synched to disk, then slows to true I/O speed. Then the whole system chugs until this write-behind has been cleared. – Steve Jessop Sep 08 '12 at 03:37
  • Of course this theory is wrong if Linux does a "proper" synch to disk on process exit – Steve Jessop Sep 08 '12 at 03:39
  • Linux does not necessarily synch to disk on process exit. – Alex Brown Sep 08 '12 at 03:45
  • @Alex: thanks, that's what I suspected but I wasn't sure. That's why you don't want to trip over the power cable. – Steve Jessop Sep 08 '12 at 03:56

5 Answers5

5

Can there be a malloc()/free() mismatch that will lose memory permanently, i.e. even after the process completes?

No, and free, and even are harmless in this respect, and when the process terminates the OS (SUSE Linux in this case) claims all their memory back (unless it's shared with some other process that's still running).

What other things in a C program (not C++) can cause permanent memory loss, i.e. after the program completes, and even the terminal window closes? Only a reboot brings the memory back. I've read other posts about files not being closed causing problems, but, I don't think I have that problem.

Like malloc/free and mmap, files opened by the process are automatically closed by the OS.

There are a few things which cause permanent memory leaks like big pages but you would certainly know about it if you were using them. Apart from that, no.

However, if you define memory loss as memory not marked 'free' immediately, then a couple of things can happen.

  1. Writes to disk or mmap may be cached for a while in RAM. The OS must keep the pages around until it synchs them back to disk.
  2. Files READ by the process may remain in memory if the OS has nothing else to use that RAM for right now - on the reasonable assumption that it might need them soon and it's quicker to read the copy that's already in RAM. Again, if the OS or another process needs some of that RAM, it can be discarded instantly.

Note that as someone who paid for all my RAM, I would rather the OS used ALL of it ALL the time, if it helps in even the smallest way. Free RAM is wasted RAM.

The main problem with having little free RAM is when it is overcommitted, which is to say there are more processes (and the OS) asking for or using RAM right now than is available on the system. It sounds like you are using about 4Gb of RAM in your processes, which might be a problem - (and remember the OS needs a good chunk too. But it sounds like you have plenty of RAM! Try running half the number of processes and see if it gets better.

Sometimes a memory leak can cause temporary overcommitment - it's a good idea to look into that. Try plotting the memory use of your program over time - if it rises continuously, then it may well be a leak.

Note that forking a process creates a copy that shares the memory the original allocated - until both are closed or one of them 'exec's. But you aren't doing that.

Is it valid to be looking at top and free for the memory statistics, i.e. do they accurately describe the memory situation? They do seem to correspond to the slowness of the program.

Yes, top and ps are perfectly reasonable ways to look at memory, in particular observe the RES field. Ignore the VIRT field for now. In addition:

To see what the whole system is doing with memory, run:

vmstat 10

While your program is running and for a while after. Look at what happens to the ---memory--- columns.

In addition, after your process has finished, run 

cat /proc/meminfo

And post the results in your question.

If the program only shows a 4% memory usage, will something like valgrind find this problem?

Probably, but it can be extremely slow, which might be impractical in this case. There are plenty of other tools which can help such as electricfence and others which do not slw your program down noticeably. I've even rolled my own in the past.

Alex Brown
  • 41,819
  • 10
  • 94
  • 108
  • No forking or exec'ing. just threads. I still see the problem when reducing the thread count down to something like 4, and I have 16GB of RAM, so there should be enough for what I'm trying to do with 32 threads. – Keith Hogan Sep 08 '12 at 03:50
  • ok, let me take a look at my linux box, we should be able to find out where your RAM has gone after termination. – Alex Brown Sep 08 '12 at 03:52
  • I've run it with various thread counts from 1 to 32. At some time, regardless of the thread count, it grinds to a halt. Maybe it's something like too many open file handles or some other such system resource. – Keith Hogan Sep 08 '12 at 03:55
  • could be - but you did say the program terminates? Try out the vmstat and meminfo spells in my answer. – Alex Brown Sep 08 '12 at 04:02
3

malloc()/free() work on the heap. This memory is guaranteed to be released to the OS when the process terminates. It is possible to leak memory even after the allocating process terminates using certain shared memory primitives (e.g. System V IPC). However, I don't think any of this is directly relevant.

Stepping back a bit, here's output from a lightly-loaded Linux server:

$ uptime
 03:30:56 up 72 days,  8:42,  2 users,  load average: 0.06, 0.17, 0.27
$ free -m
             total       used       free     shared    buffers     cached
Mem:         24104      23452        652          0      15821        978
-/+ buffers/cache:       6651      17453
Swap:         3811          5       3806

Oh no, only 652 MB free! Right? Wrong.

Whenever Linux accesses a block device (say, a hard drive), it looks for any unused memory, and stores a copy of the data there. After all, why not? The data's already in RAM, some program clearly wanted that data, and RAM that's unused can't do anyone any good. If a program comes along and asks for more memory, the cached data is discarded to make room -- until then, might as well hang onto it.

The key to this free output is not the first line, but the second. Yes, 23.4 GB of RAM is being used -- but 17.4 GB is available for programs that want it. See Help! Linux ate my RAM! for more.

I can't say why the program is getting slower, but having the "free memory" metric steadily drop down to nothing is entirely normal and not the cause.

willglynn
  • 11,210
  • 48
  • 40
  • Then, maybe I'm looking at another problem, like file handles being used up or something like that? – Keith Hogan Sep 08 '12 at 03:53
  • File handles are also reclaimed by the OS when the process terminates, again with the exception of weird interprocess stuff. Specifics on the tool involved might help focus speculation. – willglynn Sep 08 '12 at 03:57
  • I don't agree that the second line is more relevant. The first line is the useful one. Reasoning like you suggest about the second line is very misleading. 17.4GB is not available for programs that want it because if that much memory was used by programs, there would be nothing left over for a disk cache and performance would tank. It's the first line you should focus on, 652MB is free, and that's perfectly fine. 15GB is buffers, and you need to know whether that's an adequate disk cache for your load to know wheter this node is okay on memory or not. – David Schwartz Sep 08 '12 at 08:51
1

The operating system only makes as much memory free as it absolutely needs. Making memory free is wasted effort if the memory is later used normally -- it's more efficient to just directly transition the memory from one use to another than to make the memory free just to have to make it unfree later.

The only thing the system needs free memory for is operations that require memory that can't switch used memory from one purpose to another. This is a very small set of unusual operations such as servicing network interrupts.

If you type this command sysctl vm.min_free_kbytes, the system will tell you the number of KB it needs free. It's likely less than 100MB. So having any amount more than that free is perfectly fine.

If you want more of your memory free, remove it from the computer. Otherwise, the operating system assumes that there is zero cost to using it, and thus zero benefit to making it free.

For example, consider the data you wrote to disk. The operating system could make the memory that was holding that data free. But that's a double loss. If the data you wrote to disk is later read, it will have to read it from disk rather than just grabbing it from memory. And if that memory is later needed for some other purpose, it will just have to undo all the work it went through making it free. Yuck. So if the system doesn't absolutely need free memory, it won't make it free.

David Schwartz
  • 179,497
  • 17
  • 214
  • 278
  • But when I run my fragmentation test program, and it completes, or I stop it part way through (^C) with memory left allocated and not freed, the top/free numbers always snap back to where they were when I started the program. What would be the difference with the real program? – Keith Hogan Sep 08 '12 at 03:37
  • @user1426181: the fragmentation test program isn't hitting disk, right? The OS continues "using" memory that ever had disk cache in it, until needed, just in case it's useful. But the mechanism by which it frees the `malloc`ed memory when the process exits, is by unmapping the process's virtual address space (which it would have to do anyway, so basically that's costless). That memory does get marked as free, or at least that's my limited understanding. – Steve Jessop Sep 08 '12 at 03:59
  • Right, the fragmentation program doesn't hit the disk. – Keith Hogan Sep 08 '12 at 04:03
  • @user1426181: In that case, the OS has no choice. If the memory cannot be referenced in any way, it must make it free. However, if the memory contains copies of data that are also on disk, then the memory could be referenced by an attempted to read that data from the disk. (There are other ways memory can be referencable, but that's likely the one that's relevant here.) – David Schwartz Sep 08 '12 at 04:10
0

My guess would be the problem is not in your program, but in the operating system. The OS keeps a cache of recently used files in memory on the assumption that you are going to access them again. It does not know with certainty what files are going to be needed, so it can end up deciding to keep the wrong ones at the expense of the ones you wish it was keeping.

It may be keeping the output files of the first run cached when you do your second run, which prevents it from effectively using the cache on the second run. You can test this theory by deleting all files from the first run (which should free them from cache) and seeing if that makes the second run go faster.

If that doesn't work, try deleting all the input files for the first run as well.

J Nance
  • 21
  • 1
-1

Answers

  1. Yes there is no requirement in C or C++ to release memory that is not freed back to the OS
  2. Do you have memory mapped files, open file handles for deleted files etc. Linux will not delete a file until all references to is a deallocated. Also linux will cache the file in memory in case it needs to be read again - file cache memory usage can be ignored as the OS will deal with it
  3. No
  4. Maybe valgrind will highlight cases where memory is not
Adrian Cornish
  • 23,227
  • 13
  • 61
  • 77
  • There's no requirement in C or C++ to recover leaked `malloc`ed memory at process exit, but there's probably such a requirement in Posix, isn't there? Question is tagged and mentioned SUSE just as much as C. – Steve Jessop Sep 08 '12 at 03:15
  • For #3, if top/free aren't valid, what is the best way to determine what memory is available? – Keith Hogan Sep 08 '12 at 03:16
  • @SteveJessop So if you run this on AdrianOS 1.0 then it does not matter, it is bad to rely on external factors – Adrian Cornish Sep 08 '12 at 03:16
  • @user1426181 Have a look at the output from `cat /proc/slabinfo` but there are all sort of caveats - ie a shared library can be reported by every process as memory in use but it is only loaded once – Adrian Cornish Sep 08 '12 at 03:17
  • He's not "relying on external factors", he's asking for an explanation of the behavior of code he believes to be leaking memory. The behavior was observed on SUSE, not on AdrianOS, so I don't quite see the relevance of information about how it might behave on AdrianOS. Your other points all seem relevant to Posix/Linux/SUSE. – Steve Jessop Sep 08 '12 at 03:17
  • @SteveJessop Simply because C behaves as it does on ANY operating system in the world - that is the limit of the standard - what the OS does is not relevant to this language - there is a core problem - SuSe is more likely to be ok than the OP's code – Adrian Cornish Sep 08 '12 at 03:20
  • So, *if* he had observed this same behavior on AdrianOS, *then* he could suspect that the permanent loss of memory was due to a malloc/free leak. Since he observed it on SUSE, he can conclude that it is not due solely to a malloc/free leak in his code, but must be something else. Since on SUSE (unlike on AdrianOS), a malloc/free leak alone cannot account for the observed behavior. Correct? The OKness of SUSE provides us with information that the C standard does not. – Steve Jessop Sep 08 '12 at 03:23
  • @SteveJessop We can go to chat if you want, but a simplistic example is that my code allocated a Gb of ram then frees it straight away but keeps running - where is the requirement to give this memory back to the OS – Adrian Cornish Sep 08 '12 at 03:28
  • @SteveJessop I am not explaining my point well - something that seems like a leak may not be - but whatever – Adrian Cornish Sep 08 '12 at 03:31
  • This can be totally expected to eat all the free memory on any machine `#include int main() { for(;;) { void *ptr=malloc(10000000); free(ptr); } } ` – Adrian Cornish Sep 08 '12 at 03:36
  • 1
    It doesn't on mine (Windows). 50% CPU (that's one core of 2), 500k private working set, 13MB commit size. It's just using the same 10MB over and over again. Fan's got noisier, other than that it's not affecting me at all. Remove the call to `free` and it's the same, except 6MB private working set and 2GB commit size. This is on a machine with 2GB RAM, and I'm typing happily away :-) – Steve Jessop Sep 08 '12 at 03:47