2

I have very strange case of memory usage on our ubuntu server. One process (which is searchd from sphinxsearch) allocated almost all available memory, and its VSize, RSS and SHR are almost equal (about 18GB). But what makes me really astonished, is that command free treats most of this memory as "cached" - which I always thought is "kernel owned", that is - not bound to particular process. Also, in the same time it's marked as "shared", although there're no other processes with such high memory usage.

So, free -h shows:

root@st3:/proc/31633# free -h
             total       used       free     shared    buffers     cached
Mem:           23G        22G       649M        18G        62M        18G
-/+ buffers/cache:       4.4G        19G
Swap:           0B         0B         0B

But for the searchd process we have:

VmPeak: 20451512 kB
VmSize: 20413352 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:  20325488 kB
VmRSS:  20287332 kB
VmData:  1344768 kB
VmStk:       136 kB
VmExe:      4268 kB
VmLib:     16204 kB
VmPTE:     39924 kB
VmSwap:        0 kB

So I can't really understand what is real usage here - it seems most of memory is used only for cache, so it shouldn't be a concern, OTOH we already encountered several failures with "Cannot allocate memory" for simple fork, so that's why I'm trying to understand it.

If you want more, here's the full meminfo from that machine, and here is the searchd process' full list of memory mappings.

And looking at the last one, I see a lot of:

7f7c905b7000-7f7c90713000 rw-s 00000000 00:05 226801755                  /dev/zero (deleted)
7f7c90713000-7f7c91fff000 rw-s 00000000 00:05 225400928                  /dev/zero (deleted)
7f7c92055000-7f7c921c7000 rw-s 00000000 00:05 226767567                  /dev/zero (deleted)
7f7c921da000-7f7c92338000 rw-s 00000000 00:05 226774289                  /dev/zero (deleted)

...so I can only guess this could be a point, that searchd is doing some clever tricks to keep memory but in the same point have it available for the system when needed. And maybe it doesn't fully work as expected. But that's only my wild guess, and I can be completely wrong here.

kompas
  • 123
  • 4

1 Answers1

1

The 'Cached' entry is counting the number of pages marked as 'shared'. The mappings given are marked as shared.

The kernel internally sees no difference in memory that is set with the shared flag and a file thats simply been catted by the operating system and stored in cache (all files in cache are effectively shared mappings).

The fact this is backed by /dev/zero is inconsequential. The reason they are shared is almost certainly because the same memory data needs to be made available to all child processes that modify the data.

From a semantics perspective it behaves like normally allocated memory (or anonymous memory) given there really is no usable file you can evict pages to (/dev/zero is a not really a file you can save into) and the pages would get moved to swap if they were not in use.

So - confusingly - the data accounts as 'cached' but actually is treated like anonymously backed memory.

You can see this exactly is the case in your meminfo:

root@st3:/proc/31633# cat /proc/meminfo 
MemTotal:       24690512 kB
...
Cached:         19504260 kB
...
Active(anon):    3734376 kB
Inactive(anon): 18973184 kB
Active(file):     227128 kB
Inactive(file):   365828 kB
...
Mapped:         19237684 kB
Shmem:          18985464 kB
...

The same behaviour occurs if you are using IPC shared memory too.

'file' really is whats file backed, 'anon' is whats not got a file backing it -- like your mappings.

Matthew Ife
  • 23,357
  • 3
  • 55
  • 72
  • "The kernel internally sees no difference in memory that is set with the shared flag and a file thats simply been catted by the operating system and stored in cache". But what about re-using this memory? For the (kernel) cached mem, when there's a request for the memory and no free available, cache pages are freed and allocated for the requesting process. But here I guess it's not the case, and this memory is really used - in the sense, that it cannot be used by any other process? So we really lack memory on this machine? – kompas Mar 09 '16 at 14:10
  • As mentioned, its counted as 'cached' (file backed) but treated as anonymous (real memory). – Matthew Ife Mar 09 '16 at 14:55
  • So it cannot be reused, and is the reason of our "Cannot allocate memory" problems? – kompas Mar 09 '16 at 15:14
  • Definitely cannot be evicted. Whether its the source of your issues I dont know, although it looks to be using a lot of the memory. – Matthew Ife Mar 09 '16 at 18:32