Out of Memory Mismatch compared to what system rerports

Question

Oct 25 07:41:32 KVM-BOX kernel: memory: usage 255216608kB, limit 255216640kB, failcnt 28058

[root@KVM-BOX ~]# free -hm
              total        used        free      shared  buff/cache   available
Mem:           282G        203G         78G        173M        955M         78G
Swap:            0B          0B          0B           0B          0B          0B 

[root@KVM-BOX ~]# cat /proc/meminfo
MemTotal:       296615980 kB
MemFree:        82997080 kB
MemAvailable:   82829388 kB

Every metric I can see suggests that there should be 296615980kb of memory available, yet OOM seems to think only 255216612kB of memory is available?

# lsmem -a 
RANGE SIZE STATE REMOVABLE BLOCK
0x0000000000000000-0x000000007fffffff 2G online no 0 
0x0000000100000000-0x000000017fffffff 2G online no 2 
0x0000000180000000-0x00000001ffffffff 2G online no 3 
0x0000000200000000-0x000000027fffffff 2G online no 4 
0x0000000280000000-0x00000002ffffffff 2G online no 5 
0x0000004800000000-0x000000487fffffff 2G online no 144 

Memory block size: 2G 
Total online memory: 288G 
Total offline memory: 0B

Answer 1

Read the rest of the log lines you omitted, and check if the kill message matches "Out of memory" or "Memory cgroup out of memory". In the latter case, memory limits are a resource control feature on cgroups.

OOM killer can trigger when the cgroup limit is reached. When a group's quota is exhausted, the least bad option is the same as with the full system: kill something. Free up memory in that group without endangering the system as a whole.

Look at whatever memory cgroups you have on this host.

Particularly systemd: systemd-cgtop --order=memory

Even if there were no cgroups in play, 243 used of 282 GB is under some memory pressure. 86% utilized is getting high. Do some capacity planning. If KVM-BOX is a VM host with large memory allocated to guests, do not start more above this utilization.