Our monitoring tool regularly reports low memory for one of our servers running ZFS.
# free -g
total used free shared buff/cache available
Mem: 125 84 39 0 2 40
Swap: 49 0 49
After googling I found that ZFS is quite memory hungry with it's cache. However it uses "only" 44.9 GB (or do I read the arc_summary incorrectly:
# arc_summary
...
ARC size (current): 85.2 % 53.5 GiB
Target size (adaptive): 85.8 % 53.9 GiB
Min size (hard limit): 6.2 % 3.9 GiB
Max size (high water): 16:1 62.8 GiB
...
So:
125 GB Memory Total
- 40 GB Available
- 54 GB ZFS Cache
- 6 GB Processes displayed by ps (vsz)
=====
25 GB ????
As requested: (cat /proc/meminfo):
MemTotal: 131665836 kB
MemFree: 41276812 kB
MemAvailable: 42553236 kB
Buffers: 0 kB
Cached: 77304 kB
SwapCached: 24236 kB
Active: 60604 kB
Inactive: 83652 kB
Active(anon): 12712 kB
Inactive(anon): 66792 kB
Active(file): 47892 kB
Inactive(file): 16860 kB
Unevictable: 11948 kB
Mlocked: 11948 kB
SwapTotal: 52428796 kB
SwapFree: 52373500 kB
Dirty: 0 kB
Writeback: 464 kB
AnonPages: 55112 kB
Mapped: 56168 kB
Shmem: 2296 kB
KReclaimable: 2084224 kB
Slab: 65863304 kB
SReclaimable: 2084224 kB
SUnreclaim: 63779080 kB
KernelStack: 13856 kB
PageTables: 3964 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 118261712 kB
Committed_AS: 361476 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 2948856 kB
VmallocChunk: 0 kB
Percpu: 31232 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 88585508 kB
DirectMap2M: 43186176 kB
DirectMap1G: 4194304 kB
systemd-cgtop -m -n 1
Control Group Tasks %CPU Memory Input/s Output/s
/ 787 - 86.1G - -
system.slice 98 - 2.0G - -
system.slice/cron.service 1 - 1.9G - -
system.slice/systemd-journald.service 1 - 32.5M - -
The rest is under 30 MB
slabtop --once
Active / Total Objects (% used) : 99549552 / 109644976 (90.8%)
Active / Total Slabs (% used) : 3498896 / 3498896 (100.0%)
Active / Total Caches (% used) : 127 / 196 (64.8%)
Active / Total Size (% used) : 61511851.41K / 65204041.81K (94.3%)
Minimum / Average / Maximum Object : 0.01K / 0.59K / 16.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
16127096 14742570 91% 0.50K 503977 32 8063632K kmalloc-512
15970928 14903038 93% 0.38K 380264 42 6084224K dmu_buf_impl_t
15562646 14573457 93% 0.97K 483843 33 15482976K dnode_t
13621377 13469293 98% 0.24K 412769 33 3302152K sa_cache
13543561 13468664 99% 1.08K 649473 29 20783136K zfs_znode_cache
10347792 10347792 100% 0.19K 246376 42 1971008K dentry
5660031 5660011 99% 0.31K 110981 51 1775696K arc_buf_hdr_t_full
5105792 3032351 59% 0.03K 39889 128 159556K kmalloc-32
3003328 2081202 69% 0.06K 46927 64 187708K kmalloc-64
2873949 2097731 72% 0.10K 73691 39 294764K abd_t
2225216 272877 12% 0.25K 69538 32 556304K kmalloc-256
672042 670950 99% 0.09K 16001 42 64004K kmalloc-96
619191 488804 78% 0.08K 12141 51 48564K arc_buf_t
565984 538604 95% 1.00K 17687 32 565984K kmalloc-1k
485063 460395 94% 8.00K 122762 4 3928384K kmalloc-8k
483359 463686 95% 16.00K 244241 2 7815712K zio_buf_comb_16384
418272 215389 51% 1.00K 13071 32 418272K zio_buf_comb_1024
392576 392576 100% 0.06K 6134 64 24536K kmalloc-rcl-64
326080 326080 100% 0.12K 10190 32 40760K scsi_sense_cache
256494 44775 17% 0.19K 6107 42 48856K kmalloc-192
189224 189224 100% 0.07K 3379 56 13516K Acpi-Operand
I need to lower memory usage. I could reduce the zfs cache, but that it not the only memory hog, so it might make more sense to reduce the 35 unknown GB.
How can I find out what is using the memory?
