Slab SReclaimable memory can not be recycled?

Question

CentOS Linux release 7.2.1511 (Core)

Linux version 3.10.0-514.26.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP TueJul 4 15:04:05 UTC 2017

/proc/meminfo:

MemTotal:       16267428 kB
MemFree:          237816 kB
MemAvailable:    7501712 kB
Buffers:           18076 kB
Cached:           745340 kB
SwapCached:            0 kB
Active:          5015316 kB
Inactive:         152100 kB
Active(anon):    4404088 kB
Inactive(anon):      972 kB
Active(file):     611228 kB
Inactive(file):   151128 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:              1928 kB
Writeback:             0 kB
AnonPages:       4404052 kB
Mapped:            36320 kB
Shmem:              1008 kB
Slab:           10579260 kB
SReclaimable:    6839864 kB
SUnreclaim:      3739396 kB
KernelStack:       19232 kB
PageTables:        25760 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     8133712 kB
Committed_AS:    7992196 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       94920 kB
VmallocChunk:   34359635708 kB
HardwareCorrupted:     0 kB
AnonHugePages:   2297856 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      161664 kB
DirectMap2M:    10323968 kB
DirectMap1G:     8388608 kB

slabtop:

 Active / Total Objects (% used)    : 18223363 / 42966058 (42.4%)
 Active / Total Slabs (% used)      : 1183671 / 1183671 (100.0%)
 Active / Total Caches (% used)     : 73 / 95 (76.8%)
 Active / Total Size (% used)       : 4513721.33K / 10427564.51K (43.3%)
 Minimum / Average / Maximum Object : 0.01K / 0.24K / 8.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
6763716 6763212  99%    0.11K 187881       36    751524K sysfs_dir_cache
5604032 949314  16%    0.06K  87563       64    350252K kmalloc-64
4202094  67116   1%    0.04K  41197      102    164788K ext4_extent_status
3893484 2373049  60%    0.19K 185404       21    741616K dentry
3748191 1802716  48%    0.58K 138832       27   2221312K inode_cache
3251724 987321  30%    0.09K  77422       42    309688K kmalloc-96
2611301 1924963  73%    0.57K  93416       28   1494656K radix_tree_node
2590224 764829  29%    0.10K  66416       39    265664K buffer_head
2042465 284009  13%    0.05K  24029       85     96116K shared_policy_node
1802221 613054  34%    1.01K  58287       31   1865184K ext4_inode_cache
1263674 182269  14%    0.31K  50548       25    404384K nf_conntrack_ffffffff81aa0e80
1251789 210295  16%    0.19K  59609       21    238436K kmalloc-192
726016 686721  94%    0.03K   5672      128     22688K kmalloc-32
712992   9160   1%    0.50K  22281       32    356496K kmalloc-512
591360  43401   7%    0.12K  18480       32     73920K kmalloc-128
579564   4356   0%    0.11K  16099       36     64396K jbd2_journal_head
310514   4893   1%    2.00K  19576       16    626432K kmalloc-2048
183680 181248  98%    0.06K   2870       64     11480K kmem_cache_node
181936 180969  99%    0.25K   5686       32     45488K kmem_cache
130254   1632   1%    0.04K   1277      102      5108K Acpi-Namespace
 84512  19793  23%    1.00K   2641       32     84512K kmalloc-1024
 83312   2464   2%    0.25K   2608       32     20864K dquot
 80224  12022  14%    0.25K   2574       32     20592K kmalloc-256
 53538   2009   3%    1.94K   3347       16    107104K TCP
 39490  15690  39%    4.00K   5106        8    163392K kmalloc-4096
 28800    860   2%    1.56K   1440       20     46080K mm_struct
 23808  20992  88%    0.02K     93      256       372K kmalloc-16

To the problem:

I running some docker containers on this host, set mem limit about 13g(5g actually used). I want to start another java process, but killed by oom killer. SReclaimable of Slab can not be freed.

Things I tried

echo 3 > /proc/sys/vm/drop_caches

Answer 1

What you have

From meminfo, that is 7.2 GB of MemAvailable. Consider this a relatively safe capacity planning limit; think carefully before adding more than 7 GB.

No swap. Which is fine. However, there is no opportunity to use swap properly, as a low volume page out. So reclaim will be more difficult, slightly increasing the risk of OOM if running fully utilized.

Recommendations

Move workloads around until you have 13 GB MemAvailable, or find a different host that has this much.

Double check your estimates as to the memory usage of this Java app. Heap is only the largest consumer for a JVM, there are other allocations. And, if there are other processes in the container, add those too. Consider rounding up your first estimate, for starters. Hard limits like cgroup memory.max must be bigger than this.

---

echo 3 > /proc/sys/vm/drop_caches

Do not drop_caches outside of benchmarks. It cannot help. All it does it throw away warm caches that may need to be re-read from storage. SReclaimable is already a part of MemAvailable, as memory that will be reclaimed first.