3

The following C program allocates slices of 10MB of memory until it is killed by the Out of Memory killer.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

int main() {
  int counter = 0;
  int slice_size = 10*1000*1000;
  char c = 0;

  while (1) {
    char *assigned = malloc(slice_size);
    memset(assigned, c, slice_size);
    counter++;
    printf("Managed to assign %d times %d\n", counter, slice_size);
    usleep(100000)
  }
}

Given that free -m (ran immediately after the program ran) reports 6450 MB of free memory

              total       used       free     shared    buffers     cached
Mem:          7583       1595       5988          0          0        462
-/+ buffers/cache:       1132       6450
Swap:            0          0          0

why does this program only manage to assign 623 times 10MB before it is killed?

Managed to assign 1 times 10000000
[..]
Managed to assign 622 times 10000000
Managed to assign 623 times 10000000
Killed

There seems to be a hole of about 220 MB between what I thought I could malloc and what I could actually malloc. It's not kernel memory: cat /proc/meminfo immediately after the program ran:

MemTotal:        7765376 kB
MemFree:         6131572 kB
Buffers:             568 kB
Cached:           473472 kB
SwapCached:            0 kB
Active:          1157360 kB
Inactive:         265020 kB
Active(anon):    1157152 kB
Inactive(anon):   207900 kB
Active(file):        208 kB
Inactive(file):    57120 kB
Unevictable:          32 kB
Mlocked:              32 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:        948992 kB
Mapped:            37272 kB
Shmem:            415980 kB
Slab:              69556 kB
SReclaimable:      29180 kB
SUnreclaim:        40376 kB
KernelStack:        5152 kB
PageTables:        42932 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     3882688 kB
Committed_AS:    5347532 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      393956 kB
VmallocChunk:   34359336444 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      122368 kB
DirectMap2M:     7985152 kB

Some possible explanations:

  • There is memory that could be freed, and will be freed, if memory pressure slowly increases, but it hasn't yet been freed and isn't freed rapidly enough to cope with this program. I added the usleep to try to provide some time. It doesn't seem to matter

  • The memory is being assigned, but the OOM killer prevents the shell from printing what it received from printf. Again, I'd think the usleep provides plenty of time to allow that.

OOM killer output:

[ 4153.253325] indicator-cpufr invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
[ 4153.253330] indicator-cpufr cpuset=/ mems_allowed=0
[ 4153.253332] Pid: 9282, comm: indicator-cpufr Tainted: PF          O 3.8.0-34-generic #49-Ubuntu
[ 4153.253334] Call Trace:
[ 4153.253340]  [snip: the usual]
[ 4153.253399] Mem-Info:
[ 4153.253400] Node 0 DMA per-cpu:
[ 4153.253402] CPU    0: hi:    0, btch:   1 usd:   0
[ 4153.253404] CPU    1: hi:    0, btch:   1 usd:   0
[ 4153.253405] CPU    2: hi:    0, btch:   1 usd:   0
[ 4153.253406] CPU    3: hi:    0, btch:   1 usd:   0
[ 4153.253407] Node 0 DMA32 per-cpu:
[ 4153.253409] CPU    0: hi:  186, btch:  31 usd:   0
[ 4153.253411] CPU    1: hi:  186, btch:  31 usd:   0
[ 4153.253412] CPU    2: hi:  186, btch:  31 usd:   0
[ 4153.253413] CPU    3: hi:  186, btch:  31 usd:   0
[ 4153.253414] Node 0 Normal per-cpu:
[ 4153.253416] CPU    0: hi:  186, btch:  31 usd:   0
[ 4153.253417] CPU    1: hi:  186, btch:  31 usd:   0
[ 4153.253418] CPU    2: hi:  186, btch:  31 usd:  16
[ 4153.253420] CPU    3: hi:  186, btch:  31 usd:   0
[ 4153.253424] active_anon:1799845 inactive_anon:59155 isolated_anon:0
[ 4153.253424]  active_file:36 inactive_file:0 isolated_file:0
[ 4153.253424]  unevictable:8 dirty:0 writeback:0 unstable:0
[ 4153.253424]  free:25796 slab_reclaimable:7380 slab_unreclaimable:10392
[ 4153.253424]  mapped:10662 shmem:115550 pagetables:14191 bounce:0
[ 4153.253424]  free_cma:0
[ 4153.253428] Node 0 DMA free:15884kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15644kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[ 4153.253432] lowmem_reserve[]: 0 2761 7776 7776
[ 4153.253435] Node 0 DMA32 free:43900kB min:23952kB low:29940kB high:35928kB active_anon:2571536kB inactive_anon:40804kB active_file:4kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2827940kB managed:2629680kB mlocked:0kB dirty:0kB writeback:0kB mapped:16236kB shmem:40804kB slab_reclaimable:8kB slab_unreclaimable:76kB kernel_stack:0kB pagetables:5032kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:67 all_unreclaimable? yes
[ 4153.253439] lowmem_reserve[]: 0 0 5014 5014
[ 4153.253442] Node 0 Normal free:43400kB min:43496kB low:54368kB high:65244kB active_anon:4627844kB inactive_anon:195816kB active_file:140kB inactive_file:28kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:5134752kB managed:5085580kB mlocked:32kB dirty:0kB writeback:0kB mapped:26412kB shmem:421396kB slab_reclaimable:29512kB slab_unreclaimable:41476kB kernel_stack:5224kB pagetables:51732kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:348 all_unreclaimable? yes
[ 4153.253447] lowmem_reserve[]: 0 0 0 0
[ 4153.253450] Node 0 DMA: 1*4kB (U) 1*8kB (U) 0*16kB 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15884kB
[ 4153.253460] Node 0 DMA32: 55*4kB (UEM) 57*8kB (UEM) 11*16kB (UEM) 10*32kB (UEM) 4*64kB (UEM) 2*128kB (E) 4*256kB (UEM) 3*512kB (EM) 3*1024kB (EM) 6*2048kB (UE) 6*4096kB (ER) = 44180kB
[ 4153.253472] Node 0 Normal: 189*4kB (UEM) 300*8kB (UEM) 94*16kB (UEM) 151*32kB (UEM) 138*64kB (E) 102*128kB (EM) 32*256kB (E) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 43668kB
[ 4153.253483] 115782 total pagecache pages
[ 4153.253484] 0 pages in swap cache
[ 4153.253486] Swap cache stats: add 0, delete 0, find 0/0
[ 4153.253486] Free swap  = 0kB
[ 4153.253487] Total swap = 0kB
[ 4153.269100] 2057712 pages RAM
[ 4153.269103] 116530 pages reserved
[ 4153.269104] 1123397 pages shared
[ 4153.269105] 1879649 pages non-shared
[ 4153.269106] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[ 4153.269115] [  491]     0   491     3818       55      13        0             0 upstart-file-br
[ 4153.269119] [  551]     0   551     4462      204      14        0             0 upstart-udev-br
[ 4153.269121] [  557]     0   557     5437      197      16        0         -1000 udevd
[ 4153.269125] [ 1005]     0  1005     3815       50      12        0             0 upstart-socket-
[ 4153.269128] [ 1178]     0  1178    13064      150      30        0         -1000 sshd
[ 4153.269131] [ 1193]   102  1193     6278      389      16        0             0 dbus-daemon
[ 4153.269133] [ 1232]     0  1232    20853      207      45        0             0 modem-manager
[ 4153.269135] [ 1238]     0  1238     4944      218      13        0             0 bluetoothd
[ 4153.269138] [ 1241]   101  1241    61866      160      23        0             0 rsyslogd
[ 4153.269140] [ 1324]     0  1324     3958       41      13        0             0 getty
[ 4153.269143] [ 1332]     0  1332     3958       43      13        0             0 getty
[ 4153.269145] [ 1340]     0  1340     3958       41      14        0             0 getty
[ 4153.269148] [ 1341]     0  1341     3958       41      13        0             0 getty
[ 4153.269150] [ 1344]     0  1344     3958       41      13        0             0 getty
[ 4153.269153] [ 1386]     0  1386     1128       73       7        0             0 acpid
[ 4153.269155] [ 1387]     0  1387     4782       42      13        0             0 atd
[ 4153.269158] [ 1388]     0  1388     5332       62      16        0             0 cron
[ 4153.269160] [ 1390]   107  1390     8120      113      22        0             0 avahi-daemon
[ 4153.269162] [ 1391]   107  1391     8056       64      21        0             0 avahi-daemon
[ 4153.269166] [ 1398]     0  1398     4788       58      15        0             0 irqbalance
[ 4153.269168] [ 1414]     0  1414    68879      208      37        0             0 lightdm
[ 4153.269170] [ 1418]     0  1418    17608      170      38        0             0 cups-browsed
[ 4153.269173] [ 1449]     0  1449    62804      474      52        0             0 NetworkManager
[ 4153.269175] [ 1557]   115  1557   137486    15239      75        0             0 mysqld
[ 4153.269178] [ 1597]     0  1597    49116      571      35        0             0 polkitd
[ 4153.269180] [ 1608]     0  1608    18162      383      39        0             0 cupsd
[ 4153.269183] [ 1881]   105  1881    71731      348      71        0             0 whoopsie
[ 4153.269185] [ 2037]   103  2037    54203      412      42        0             0 colord
[ 4153.269188] [ 2100]     0  2100   102571     1105     131        0             0 libvirtd
[ 4153.269190] [ 2154]     0  2154     9049       93      24        0             0 bumblebeed
[ 4153.269193] [ 2159]     0  2159    66152      194      29        0             0 accounts-daemon
[ 4153.269196] [ 2186]     0  2186   522940      276      58        0             0 console-kit-dae
[ 4153.269198] [ 2380]     0  2380     8010      159      20        0             0 wpa_supplicant
[ 4153.269201] [ 2392]     0  2392    55684      358      43        0             0 upowerd
[ 4153.269203] [ 2545]   117  2545     6521       63      17        0             0 dnsmasq
[ 4153.269206] [ 2570]     0  2570     5672      204      15        0             0 openvpn
[ 4153.269208] [ 2605]     0  2605     5669      203      16        0             0 openvpn
[ 4153.269211] [ 2631]     0  2631     5672      204      15        0             0 openvpn
[ 4153.269213] [ 2657]     0  2657     5672      203      16        0             0 openvpn
[ 4153.269215] [ 2673]     0  2673     5672      205      15        0             0 openvpn
[ 4153.269218] [ 2690]     0  2690     5672      207      16        0             0 openvpn
[ 4153.269220] [ 2708]     0  2708     5638      178      16        0             0 openvpn
[ 4153.269222] [ 2733]     0  2733     5669      206      16        0             0 openvpn
[ 4153.269225] [ 2763]     0  2763     5672      208      16        0             0 openvpn
[ 4153.269227] [ 2784]     0  2784     5672      207      16        0             0 openvpn
[ 4153.269230] [ 2958]   111  2958    42230       61      19        0             0 rtkit-daemon
[ 4153.269232] [ 3065]     0  3065    20401      648      44        0             0 apache2
[ 4153.269235] [ 3206]     0  3206    67222     1719      33        0             0 fail2ban-server
[ 4153.269237] [ 3246]     0  3246     3958       41      13        0             0 getty
[ 4153.269240] [ 3632]     0  3632     2561      578       9        0             0 dhclient
[ 4153.269243] [ 3675] 65534  3675     7226       63      19        0             0 dnsmasq
[ 4153.269245] [ 4033]     0  4033    90143      492      43        0             0 udisksd
[ 4153.269248] [ 6118]    33  6118    20279      577      42        0             0 apache2
[ 4153.269251] [ 6122]    33  6122    92642      644      72        0             0 apache2
[ 4153.269253] [ 6123]    33  6123    92642      644      72        0             0 apache2
[ 4153.269256] [ 7392]     0  7392     5436      199      15        0         -1000 udevd
[ 4153.269260] [ 8939]     0  8939    70211    12800     123        0             0 Xorg
[ 4153.269262] [ 9029]     0  9029    41431      220      50        0             0 lightdm
[ 4153.269264] [ 9128]  1000  9128    73124      192      38        0             0 gnome-keyring-d
[ 4153.269267] [ 9138]  1000  9138   105814      690      98        0             0 gnome-session
[ 4153.269269] [ 9190]  1000  9190     3154       80       9        0             0 ssh-agent
[ 4153.269272] [ 9191]  1000  9191     4092       54      12        0             0 gpg-agent
[ 4153.269274] [ 9194]  1000  9194     6117       65      17        0             0 dbus-launch
[ 4153.269277] [ 9195]  1000  9195     6571      472      17        0             0 dbus-daemon
[ 4153.269279] [ 9205]  1000  9205    84354      137      32        0             0 at-spi-bus-laun
[ 4153.269281] [ 9209]  1000  9209     6030      120      17        0             0 dbus-daemon
[ 4153.269284] [ 9212]  1000  9212    31181      150      30        0             0 at-spi2-registr
[ 4153.269286] [ 9230]  1000  9230   205152     1719     167        0             0 gnome-settings-
[ 4153.269289] [ 9233]  1000  9233    90808      598     111        0             0 pulseaudio
[ 4153.269292] [ 9239]  1000  9239    24569      179      47        0             0 gconf-helper
[ 4153.269294] [ 9241]  1000  9241    13472      423      32        0             0 gconfd-2
[ 4153.269296] [ 9243]  1000  9243    48067      151      30        0             0 gvfsd
[ 4153.269299] [ 9247]  1000  9247    84194      141      34        0             0 gvfsd-fuse
[ 4153.269301] [ 9263]  1000  9263   264259    10791     250        0             0 compiz
[ 4153.269303] [ 9269]  1000  9269    44594      201      24        0             0 dconf-service
[ 4153.269305] [ 9276]  1000  9276   165643     4180     169        0             0 blueman-applet
[ 4153.269308] [ 9279]  1000  9279   169464     1549     138        0             0 nm-applet
[ 4153.269310] [ 9280]  1000  9280   238670     4329     217        0             0 nautilus
[ 4153.269312] [ 9281]  1000  9281   121466      617      96        0             0 gnome-fallback-
[ 4153.269315] [ 9282]  1000  9282   135466     3534     126        0             0 indicator-cpufr
[ 4153.269317] [ 9283]  1000  9283   142378    18295     141        0             0 skype
[ 4153.269319] [ 9284]  1000  9284     5065       54      16        0             0 syndaemon
[ 4153.269322] [ 9285]  1000  9285   120883     1019     109        0             0 indicator-multi
[ 4153.269324] [ 9288]  1000  9288   212039     4308     241        0             0 pidgin
[ 4153.269327] [ 9292]  1000  9292    84593      608      91        0             0 polkit-gnome-au
[ 4153.269329] [ 9304]  1000  9304    52477      426      39        0             0 gvfs-udisks2-vo
[ 4153.269332] [ 9312]  1000  9312    70334      154      38        0             0 gvfs-afc-volume
[ 4153.269334] [ 9319]  1000  9319    46798      131      28        0             0 gvfs-mtp-volume
[ 4153.269336] [ 9325]  1000  9325    49839      182      35        0             0 gvfs-gphoto2-vo
[ 4153.269339] [ 9331]  1000  9331   541243    15362     243        0             0 dropbox
[ 4153.269341] [ 9369]  1000  9369    68153      175      34        0             0 gvfsd-trash
[ 4153.269344] [ 9395]  1000  9395    13434      115      30        0             0 obex-data-serve
[ 4153.269346] [ 9416]  1000  9416    87300      609      97        0             0 notify-osd
[ 4153.269349] [ 9423]  1000  9423   125419      740     104        0             0 bamfdaemon
[ 4153.269352] [ 9484]  1000  9484    66501      146      32        0             0 gvfsd-burn
[ 4153.269355] [ 9488]  1000  9488     1111       25       8        0             0 sh
[ 4153.269357] [ 9489]  1000  9489    84483      744      92        0             0 gtk-window-deco
[ 4153.269359] [ 9492]  1000  9492   166134     3518     161        0             0 unity-panel-ser
[ 4153.269362] [ 9494]  1000  9494   172370     1883      47        0             0 hud-service
[ 4153.269364] [ 9514]  1000  9514   118985      360      81        0             0 indicator-bluet
[ 4153.269366] [ 9516]  1000  9516    95283      313      74        0             0 indicator-sync-
[ 4153.269369] [ 9518]  1000  9518   151967      543     114        0             0 indicator-datet
[ 4153.269371] [ 9523]  1000  9523   115774      714     116        0             0 indicator-print
[ 4153.269374] [ 9525]  1000  9525   135487      411      87        0             0 indicator-sessi
[ 4153.269376] [ 9527]  1000  9527   133932      418      78        0             0 indicator-messa
[ 4153.269378] [ 9534]  1000  9534   133075      373      80        0             0 indicator-appli
[ 4153.269381] [ 9535]  1000  9535   144953      509     100        0             0 indicator-sound
[ 4153.269383] [ 9635]  1000  9635   151805     2306     138        0             0 gnome-terminal
[ 4153.269386] [ 9642]  1000  9642   139764     2118      68        0             0 unity-applicati
[ 4153.269388] [ 9644]  1000  9644   167547      365      57        0             0 unity-files-dae
[ 4153.269391] [ 9661]  1000  9661     3709       41      13        0             0 gnome-pty-helpe
[ 4153.269393] [ 9662]  1000  9662     6237     1105      18        0             0 bash
[ 4153.269396] [ 9829]  1000  9829   146945     2246     105        0             0 unity-scope-gdr
[ 4153.269398] [ 9840]  1000  9840     6237     1105      17        0             0 bash
[ 4153.269401] [ 9843]  1000  9843     3493       38      13        0             0 optirun
[ 4153.269403] [ 9844]     0  9844    35523     6859      67        0             0 Xorg
[ 4153.269405] [ 9845]  1000  9845    43951     1981      49        0             0 screenclone
[ 4153.269408] [ 9860]  1000  9860    86292      623      93        0             0 gnome-screensav
[ 4153.269410] [ 9865]  1000  9865   724592    24172     310        0             0 chromium-browse
[ 4153.269413] [ 9871]  1000  9871    81867     1833     123        0             0 chromium-browse
[ 4153.269415] [ 9872]  1000  9872     3708       45      11        0             0 chrome-sandbox
[ 4153.269418] [ 9873]  1000  9873    86530     1711     139        0             0 chromium-browse
[ 4153.269420] [ 9874]     0  9874     5436      199      15        0         -1000 udevd
[ 4153.269423] [ 9878]  1000  9878   104963     1718     118        0             0 chromium-browse
[ 4153.269425] [ 9907]  1000  9907   292116    11200     205        0           300 chromium-browse
[ 4153.269428] [ 9918]  1000  9918   293926    13286     215        0           300 chromium-browse
[ 4153.269430] [ 9924]  1000  9924   287562     6096     191        0           300 chromium-browse
[ 4153.269433] [ 9941]  1000  9941   285277     3527     181        0           300 chromium-browse
[ 4153.269435] [ 9975]  1000  9975   289462     9824     206        0           300 chromium-browse
[ 4153.269438] [ 9982]  1000  9982   285419     3768     182        0           300 chromium-browse
[ 4153.269440] [ 9994]  1000  9994   286039     4968     187        0           300 chromium-browse
[ 4153.269443] [10009]  1000 10009   285297     3561     180        0           300 chromium-browse
[ 4153.269445] [10020]  1000 10020   288356     7152     194        0           300 chromium-browse
[ 4153.269447] [10036]  1000 10036   285329     3694     181        0           300 chromium-browse
[ 4153.269450] [10042]  1000 10042   295582    14686     222        0           300 chromium-browse
[ 4153.269452] [10048]  1000 10048   304408     4847     185        0           300 chromium-browse
[ 4153.269455] [10054]  1000 10054   287456     7668     201        0           300 chromium-browse
[ 4153.269457] [10063]  1000 10063   285511     3861     181        0           300 chromium-browse
[ 4153.269460] [10100]  1000 10100   290115     8803     205        0           300 chromium-browse
[ 4153.269462] [10107]  1000 10107   287209     5890     190        0           300 chromium-browse
[ 4153.269465] [10117]  1000 10117   143002     1878     151        0           100 chromium-browse
[ 4153.269468] [10119]  1000 10119   288513     8254     195        0           300 chromium-browse
[ 4153.269470] [10144]  1000 10144   147206      852     111        0             0 update-notifier
[ 4153.269472] [10214]  1000 10214    91188      201      47        0             0 deja-dup-monito
[ 4153.269476] [10311]  1000 10311     6237     1105      18        0             0 bash
[ 4153.269478] [10511]  1000 10511    69102     1259     115        0             0 vi
[ 4153.269480] [10543]  1000 10543   285919     4047     184        0           300 chromium-browse
[ 4153.269483] [10574]  1000 10574   288083     7169     198        0           300 chromium-browse
[ 4153.269485] [10594]  1000 10594  1499730  1484225    2906        0             0 a.out
[ 4153.269487] Out of memory: Kill process 10594 (a.out) score 765 or sacrifice child
[ 4153.269490] Killed process 10594 (a.out) total-vm:5998920kB, anon-rss:5936892kB, file-rss:8kB
[ 4153.284414] a.out: page allocation failure: order:0, mode:0x280da
[ 4153.284417] Pid: 10594, comm: a.out Tainted: PF          O 3.8.0-34-generic #49-Ubuntu
[ 4153.284418] Call Trace:
[Snip: the usual]
[ 4153.284464] Mem-Info:
[ 4153.284466] Node 0 DMA per-cpu:
[ 4153.284468] CPU    0: hi:    0, btch:   1 usd:   0
[ 4153.284469] CPU    1: hi:    0, btch:   1 usd:   0
[ 4153.284471] CPU    2: hi:    0, btch:   1 usd:   0
[ 4153.284472] CPU    3: hi:    0, btch:   1 usd:   0
[ 4153.284473] Node 0 DMA32 per-cpu:
[ 4153.284475] CPU    0: hi:  186, btch:  31 usd:   0
[ 4153.284476] CPU    1: hi:  186, btch:  31 usd:   0
[ 4153.284478] CPU    2: hi:  186, btch:  31 usd:   0
[ 4153.284479] CPU    3: hi:  186, btch:  31 usd:   0
[ 4153.284480] Node 0 Normal per-cpu:
[ 4153.284481] CPU    0: hi:  186, btch:  31 usd:   0
[ 4153.284483] CPU    1: hi:  186, btch:  31 usd:   0
[ 4153.284484] CPU    2: hi:  186, btch:  31 usd:   0
[ 4153.284485] CPU    3: hi:  186, btch:  31 usd:   0
[ 4153.284489] active_anon:1799974 inactive_anon:59155 isolated_anon:0
[ 4153.284489]  active_file:0 inactive_file:0 isolated_file:0
[ 4153.284489]  unevictable:8 dirty:0 writeback:0 unstable:0
[ 4153.284489]  free:25776 slab_reclaimable:7380 slab_unreclaimable:10392
[ 4153.284489]  mapped:10662 shmem:115550 pagetables:14191 bounce:0
[ 4153.284489]  free_cma:0
[ 4153.284493] Node 0 DMA free:15884kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15644kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[ 4153.284498] lowmem_reserve[]: 0 2761 7776 7776
[ 4153.284501] Node 0 DMA32 free:43860kB min:23952kB low:29940kB high:35928kB active_anon:2571536kB inactive_anon:40804kB active_file:4kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2827940kB managed:2629680kB mlocked:0kB dirty:0kB writeback:0kB mapped:16236kB shmem:40804kB slab_reclaimable:8kB slab_unreclaimable:76kB kernel_stack:0kB pagetables:5032kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 4153.284506] lowmem_reserve[]: 0 0 5014 5014
[ 4153.284508] Node 0 Normal free:43360kB min:43496kB low:54368kB high:65244kB active_anon:4628360kB inactive_anon:195816kB active_file:0kB inactive_file:0kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:5134752kB managed:5085580kB mlocked:32kB dirty:0kB writeback:0kB mapped:26412kB shmem:421396kB slab_reclaimable:29512kB slab_unreclaimable:41476kB kernel_stack:5224kB pagetables:51732kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 4153.284513] lowmem_reserve[]: 0 0 0 0
[ 4153.284516] Node 0 DMA: 1*4kB (U) 1*8kB (U) 0*16kB 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15884kB
[ 4153.284526] Node 0 DMA32: 67*4kB (UEM) 60*8kB (UEM) 14*16kB (UEM) 12*32kB (UEM) 5*64kB (UEM) 2*128kB (E) 3*256kB (UE) 3*512kB (EM) 3*1024kB (EM) 6*2048kB (UE) 6*4096kB (ER) = 44172kB
[ 4153.284537] Node 0 Normal: 257*4kB (UEM) 315*8kB (UEM) 90*16kB (UE) 151*32kB (UEM) 139*64kB (EM) 101*128kB (E) 31*256kB (EM) 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 43676kB
[ 4153.284548] 115654 total pagecache pages
[ 4153.284549] 0 pages in swap cache
[ 4153.284551] Swap cache stats: add 0, delete 0, find 0/0
[ 4153.284552] Free swap  = 0kB
[ 4153.284553] Total swap = 0kB
[ 4153.300780] 2057712 pages RAM
[ 4153.300783] 116530 pages reserved
[ 4153.300784] 1123391 pages shared
[ 4153.300785] 1879628 pages non-shared
voretaq7
  • 79,879
  • 17
  • 130
  • 214
Confusion
  • 191
  • 1
  • 8
  • 1
    Post the output from the OOM killer. – Michael Hampton Dec 08 '13 at 10:30
  • Added OOM killer output. What do you expect to see there? I don't see anything out of the ordinary. – Confusion Dec 08 '13 at 10:41
  • 2
    Actually that makes it obvious, the free memory was too fragmented to handle the final malloc() request. I'll post a full answer later if someone doesn't beat me to it. – Michael Hampton Dec 08 '13 at 10:46
  • Ah, fragmentation, I hadn't thought of that. That makes sense, thanks. I've now been looking at the OOM output more closely, but I don't know how to identify the fragmentation, so if you could explain that in your answer: awesome. If there's a way to reduce the fragmentation by smarter use of `malloc` (or perhaps a library that wraps `malloc`): even better. – Confusion Dec 08 '13 at 10:51
  • Please post your answer as an *answer*. (c'mon - you have enough reputation to know [how Stack Exchange sites work...](http://serverfault.com/about) – voretaq7 Dec 09 '13 at 17:04
  • I was waiting for Michael Hampton to post an answer, so I could award him the accepted answer. I have more often added a short version of the accepted answer at the top of a post, to spare people the trouble of wading through often long questions and answers to determine whether they are running into the same problem. After the answer is clear, it often turns out the question could have been much shorter. I think a short tl;dr at the top can be very helpful, even if a more complete answer is accepted. – Confusion Dec 09 '13 at 21:08
  • This isn't caused by fragmentation, unless you're using hugepages most userspace applications dont make high-order memory requests. If you look at the second example, a.out tries to allocate an 'order 0' page, which fails -- even though theres 257 available. The cause of the OOM is that `Zone: Normal` drops below its `min`. What is odd however is that if you add up all the pages from the zone data (pages managed by that zone - all the other areas that take memory) you are left with too much memory. So either theres areas of memory not being accounted for or the zoneinfo stats are broken. – Matthew Ife Dec 09 '13 at 23:40

0 Answers0