I see only 370 MB (10%) free on an AWS Amazon Linux instance:
$ free -m
total used free shared buffers cached
Mem: 3752 3709 42 0 142 185
-/+ buffers/cache: 3381 370
Swap: 0 0 0
Here's the memory usage profile over the life of the box, which has never been rebooted:
I'm puzzled, because nothing seems to be using much RAM (this is a NAT box running no server software):
top - 23:40:52 up 41 days, 13:48, 1 user, load average: 0.00, 0.01, 0.05
Tasks: 67 total, 1 running, 66 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3%us, 0.3%sy, 0.0%ni, 99.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.3%st
Mem: 3842636k total, 3799620k used, 43016k free, 146156k buffers
Swap: 0k total, 0k used, 0k free, 189716k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31600 aws 20 0 348m 96m 4524 S 0.0 2.6 10:46.35 opsworks-agent
31579 aws 20 0 413m 95m 4240 S 0.0 2.6 86:25.29 opsworks-agent
31596 aws 20 0 401m 83m 4216 S 0.0 2.2 18:43.31 opsworks-agent
31575 aws 20 0 178m 56m 2084 S 0.0 1.5 9:09.03 opsworks-agent
31924 root 20 0 109m 4176 3180 S 0.0 0.1 0:00.02 sshd
1414 newrelic 20 0 102m 3656 2236 S 1.0 0.1 49:32.82 nrsysmond
1441 root 20 0 89216 2324 640 S 0.0 0.1 1:25.47 sendmail
1214 root 20 0 237m 2188 788 S 0.0 0.1 0:02.42 rsyslogd
1448 smmsp 20 0 80676 2048 612 S 0.0 0.1 0:00.99 sendmail
1382 ntp 20 0 31312 1972 1360 S 0.0 0.1 0:04.08 ntpd
31926 ross 20 0 109m 1952 948 S 0.0 0.1 0:00.00 sshd
31927 ross 20 0 112m 1948 1580 S 0.0 0.1 0:00.01 bash
31472 root 20 0 121m 1632 928 S 0.0 0.0 4:12.10 monit
1292 root 20 0 463m 1620 988 S 0.0 0.0 0:35.47 automount
1 root 20 0 19480 1568 1248 S 0.0 0.0 0:01.36 init
1456 root 20 0 116m 1196 620 S 0.0 0.0 0:16.99 crond
32150 ross 20 0 15124 1196 920 R 0.0 0.0 0:00.04 top
1363 root 20 0 79964 1160 384 S 0.0 0.0 0:00.00 sshd
668 root 20 0 10784 996 608 S 0.0 0.0 0:00.04 udevd
1153 root 20 0 9228 968 476 S 0.0 0.0 0:02.05 dhclient
1250 dbus 20 0 21628 796 564 S 0.0 0.0 0:00.00 dbus-daemon
1196 root 16 -4 111m 776 432 S 0.0 0.0 0:05.24 auditd
1504 root 20 0 6304 728 620 S 0.0 0.0 0:00.00 agetty
785 root 20 0 10780 640 280 S 0.0 0.0 0:00.00 udevd
795 root 20 0 10780 580 220 S 0.0 0.0 0:00.00 udevd
1514 root 20 0 4180 564 476 S 0.0 0.0 0:00.00 mingetty
1523 root 20 0 4180 564 476 S 0.0 0.0 0:00.00 mingetty
1510 root 20 0 4180 560 476 S 0.0 0.0 0:00.00 mingetty
1516 root 20 0 4180 560 476 S 0.0 0.0 0:00.00 mingetty
1518 root 20 0 4180 560 476 S 0.0 0.0 0:00.00 mingetty
1521 root 20 0 4180 560 476 S 0.0 0.0 0:00.00 mingetty
1466 root 20 0 16908 368 212 S 0.0 0.0 0:00.04 atd
1413 newrelic 20 0 26692 264 64 S 0.0 0.0 0:00.00 nrsysmond
1230 root 20 0 4180 80 0 S 0.0 0.0 4:12.80 rngd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:23.38 ksoftirqd/0
4 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
6 root 20 0 0 0 0 S 0.0 0.0 0:23.99 kworker/u2:0
7 root 20 0 0 0 0 S 0.0 0.0 2:18.81 rcu_sched
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
10 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper
11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
12 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u2:1
19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 xenwatch
20 root 20 0 0 0 0 S 0.0 0.0 0:01.00 xenbus
It looks like only 10% of RAM is being used, from this output:
$ ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 19480 1568 ? Ss 2014 0:01 /sbin/init root 2 0.0 0.0 0 0 ? S 2014 0:00 [kthreadd] root 3 0.0 0.0 0 0 ? S 2014 0:23 [ksoftirqd/0] root 4 0.0 0.0 0 0 ? S 2014 0:00 [kworker/0:0] root 5 0.0 0.0 0 0 ? S 2014 0:00 [kworker/0:0H] root 6 0.0 0.0 0 0 ? S 2014 0:23 [kworker/u2:0] root 7 0.0 0.0 0 0 ? S 2014 2:18 [rcu_sched] root 8 0.0 0.0 0 0 ? S 2014 0:00 [rcu_bh] root 9 0.0 0.0 0 0 ? S 2014 0:00 [migration/0] root 10 0.0 0.0 0 0 ? S 2014 0:00 [khelper] root 11 0.0 0.0 0 0 ? S 2014 0:00 [kdevtmpfs] root 12 0.0 0.0 0 0 ? S 2014 0:00 [netns] root 13 0.0 0.0 0 0 ? S 2014 0:00 [kworker/u2:1] root 19 0.0 0.0 0 0 ? S 2014 0:00 [xenwatch] root 20 0.0 0.0 0 0 ? S 2014 0:01 [xenbus] root 112 0.0 0.0 0 0 ? S 2014 0:00 [writeback] root 115 0.0 0.0 0 0 ? SN 2014 0:00 [ksmd] root 116 0.0 0.0 0 0 ? S 2014 0:00 [kintegrityd] root 117 0.0 0.0 0 0 ? S 2014 0:00 [bioset] root 118 0.0 0.0 0 0 ? S 2014 0:00 [crypto] root 120 0.0 0.0 0 0 ? S 2014 0:00 [kblockd] root 131 0.0 0.0 0 0 ? S 2014 0:00 [md] root 218 0.0 0.0 0 0 ? S 2014 1:24 [kworker/0:1] root 233 0.0 0.0 0 0 ? S 2014 0:00 [khungtaskd] root 238 0.0 0.0 0 0 ? S 2014 0:07 [kswapd0] root 303 0.0 0.0 0 0 ? S 2014 0:00 [fsnotify_mark] root 317 0.0 0.0 0 0 ? S 2014 0:00 [kthrotld] root 323 0.0 0.0 0 0 ? S 2014 0:00 [khvcd] root 364 0.0 0.0 0 0 ? S 2014 0:00 [deferwq] root 630 0.0 0.0 0 0 ? S 2014 0:19 [jbd2/xvda1-8] root 631 0.0 0.0 0 0 ? S 2014 0:00 [ext4-rsv-conver] root 668 0.0 0.0 10784 996 ? Ss 2014 0:00 /sbin/udevd -d root 785 0.0 0.0 10780 640 ? S 2014 0:00 /sbin/udevd -d root 795 0.0 0.0 10780 580 ? S 2014 0:00 /sbin/udevd -d root 971 0.0 0.0 0 0 ? S 2014 0:00 [kauditd] root 1024 0.0 0.0 0 0 ? S 2014 0:00 [ipv6_addrconf] root 1153 0.0 0.0 9228 968 ? Ss 2014 0:02 /sbin/dhclient -q -lf /var/lib/dhclient/dhclient-eth0.leases -pf /var/run/dhclient-eth0.pid eth0 root 1196 0.0 0.0 113944 776 ? Ssl 2014 0:05 auditd root 1214 0.0 0.0 243280 2188 ? Sl 2014 0:02 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5 root 1230 0.0 0.0 4180 80 ? Ss 2014 4:12 rngd --no-tpm=1 --quiet dbus 1250 0.0 0.0 21628 796 ? Ss 2014 0:00 dbus-daemon --system root 1292 0.0 0.0 474260 1620 ? Ssl 2014 0:35 automount --pid-file /var/run/autofs.pid root 1363 0.0 0.0 79964 1160 ? Ss 2014 0:00 /usr/sbin/sshd ntp 1382 0.0 0.0 31312 1972 ? Ss 2014 0:04 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g newrelic 1413 0.0 0.0 26692 264 ? Ss 2014 0:00 /usr/sbin/nrsysmond -c /etc/newrelic/nrsysmond.cfg -p /var/run/newrelic/nrsysmond.pid newrelic 1414 0.0 0.0 105204 3360 ? Sl 2014 49:33 /usr/sbin/nrsysmond -c /etc/newrelic/nrsysmond.cfg -p /var/run/newrelic/nrsysmond.pid root 1441 0.0 0.0 89216 2324 ? Ss 2014 1:25 sendmail: accepting connections smmsp 1448 0.0 0.0 80676 2048 ? Ss 2014 0:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue root 1456 0.0 0.0 119356 1196 ? Ss 2014 0:16 crond root 1466 0.0 0.0 16908 368 ? Ss 2014 0:00 /usr/sbin/atd root 1504 0.0 0.0 6304 728 hvc0 Ss+ 2014 0:00 /sbin/agetty hvc0 38400 vt100-nav root 1510 0.0 0.0 4180 560 tty1 Ss+ 2014 0:00 /sbin/mingetty /dev/tty1 root 1514 0.0 0.0 4180 564 tty2 Ss+ 2014 0:00 /sbin/mingetty /dev/tty2 root 1516 0.0 0.0 4180 560 tty3 Ss+ 2014 0:00 /sbin/mingetty /dev/tty3 root 1518 0.0 0.0 4180 560 tty4 Ss+ 2014 0:00 /sbin/mingetty /dev/tty4 root 1521 0.0 0.0 4180 560 tty5 Ss+ 2014 0:00 /sbin/mingetty /dev/tty5 root 1523 0.0 0.0 4180 564 tty6 Ss+ 2014 0:00 /sbin/mingetty /dev/tty6 root 1738 0.0 0.0 0 0 ? S 2014 0:00 [kworker/0:1H] root 31472 0.0 0.0 124884 1632 ? Sl 2014 4:12 monit aws 31575 0.0 1.4 181608 57312 ? Sl 2014 9:09 opsworks-agent: master 31575 aws 31579 0.2 2.5 423204 98124 ? Sl 2014 86:26 opsworks-agent: keep_alive of master 31575 aws 31596 0.0 2.2 410856 85616 ? Sl 2014 18:43 opsworks-agent: statistics of master 31575 aws 31600 0.0 2.5 356956 98612 ? Sl 2014 10:46 opsworks-agent: process_command of master 31575 root 31924 0.0 0.1 111640 4176 ? Ss 23:28 0:00 sshd: ross [priv] ross 31926 0.0 0.0 111640 1952 ? S 23:28 0:00 sshd: ross@pts/0 ross 31927 0.0 0.0 115152 1952 pts/0 Ss 23:28 0:00 -bash ross 32240 0.0 0.0 117048 1208 pts/0 R+ 23:45 0:00 ps aux ross 32241 0.0 0.0 240 4 pts/0 D+ 23:45 0:00 [sed]
Am I misinterpreting these metrics? Any idea how I can determine what's using 90% of my RAM when it seems like only 10% is being used?
NOTE: I believe this is not a duplicate of this question, as my RAM is not being consumed by disk cache (according to free).
UPDATE 1 – output of slabtop:
Active / Total Objects (% used) : 15746067 / 15766743 (99.9%)
Active / Total Slabs (% used) : 750727 / 750727 (100.0%)
Active / Total Caches (% used) : 64 / 88 (72.7%)
Active / Total Size (% used) : 2978870.88K / 2981599.96K (99.9%)
Minimum / Average / Maximum Object : 0.01K / 0.19K / 8.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
15556863 15556863 100% 0.19K 740803 21 2963212K dentry
81705 73173 89% 0.10K 2095 39 8380K buffer_head
43728 43728 100% 0.95K 5466 8 43728K ext4_inode_cache
22525 22525 100% 0.05K 265 85 1060K shared_policy_node
13668 5466 39% 0.04K 134 102 536K ext4_extent_status
7098 7098 100% 0.55K 507 14 4056K inode_cache
6188 6188 100% 0.12K 182 34 728K kernfs_node_cache
5568 4768 85% 0.06K 87 64 348K kmalloc-64
4606 3204 69% 0.55K 329 14 2632K radix_tree_node
3190 3190 100% 0.18K 145 22 580K vm_area_struct
2856 1567 54% 0.19K 136 21 544K kmalloc-192
2496 2496 100% 0.06K 39 64 156K anon_vma
2048 2048 100% 0.01K 4 512 16K kmalloc-8
1680 1680 100% 0.07K 30 56 120K ext4_io_end
1536 1536 100% 0.03K 12 128 48K kmalloc-32
1280 1280 100% 0.02K 5 256 20K kmalloc-16
1232 980 79% 0.25K 77 16 308K kmalloc-256
1050 1050 100% 0.09K 25 42 100K kmalloc-96
966 966 100% 0.09K 21 46 84K ftrace_event_file
720 720 100% 0.11K 20 36 80K jbd2_journal_head
640 640 100% 0.12K 20 32 80K kmalloc-128
600 600 100% 0.62K 24 25 384K proc_inode_cache
512 512 100% 0.03K 4 128 16K jbd2_revoke_record_s
408 408 100% 0.64K 34 12 272K shmem_inode_cache
400 399 99% 0.50K 50 8 200K kmalloc-512
256 256 100% 0.02K 1 256 4K jbd2_revoke_table_s
248 218 87% 1.00K 31 8 248K kmalloc-1024
169 113 66% 0.30K 13 13 52K nf_conntrack_ffffffff81aa5940
168 168 100% 0.07K 3 56 12K eventpoll_pwq
150 136 90% 1.06K 10 15 160K signal_cache
144 127 88% 2.00K 18 8 288K kmalloc-2048
144 144 100% 0.11K 4 36 16K task_delay_info
135 135 100% 2.06K 9 15 288K sighand_cache
128 128 100% 0.06K 2 64 8K kmem_cache_node
128 128 100% 0.06K 2 64 8K ext4_free_data
120 60 50% 4.00K 15 8 480K kmalloc-4096
108 108 100% 0.62K 9 12 72K files_cache
108 108 100% 0.88K 12 9 96K mm_struct
100 89 89% 6.33K 20 5 640K task_struct
100 90 90% 0.38K 10 10 40K blkdev_requests
96 96 100% 0.25K 6 16 24K kmem_cache
96 96 100% 0.25K 6 16 24K tw_sock_TCP
90 90 100% 2.06K 6 15 192K idr_layer_cache
90 90 100% 0.13K 3 30 12K ext4_allocation_context
85 85 100% 0.05K 1 85 4K jbd2_journal_handle
73 73 100% 0.05K 1 73 4K ip_fib_trie
UPDATE 2 – output of cat /proc/meminfo:
MemTotal: 3842636 kB MemFree: 148572 kB MemAvailable: 3542752 kB Buffers: 151816 kB Cached: 247564 kB SwapCached: 0 kB Active: 513568 kB Inactive: 115604 kB Active(anon): 229796 kB Inactive(anon): 12 kB Active(file): 283772 kB Inactive(file): 115592 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 28 kB Writeback: 0 kB AnonPages: 229820 kB Mapped: 11060 kB Shmem: 16 kB Slab: 3031636 kB SReclaimable: 3024180 kB SUnreclaim: 7456 kB KernelStack: 672 kB PageTables: 4360 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 1921316 kB Committed_AS: 465112 kB VmallocTotal: 34359738367 kB VmallocUsed: 7532 kB VmallocChunk: 34359727383 kB AnonHugePages: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 3940352 kB DirectMap2M: 0 kB
