0

Hi I got OOM killer problems which caused: [mpm_prefork:error] [pid 18103] (12)Cannot allocate memory: AH00159: fork: Unable to fork new process

I had to restart the server to get it working.

In /var/log/messages near the OOM killer messages it showed the running processes.

There were 140-160 processes of httpd running.

/etc/httpd/conf/httpd.conf includes:

<IfModule prefork.c>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers       10
    MaxClients            100
    ServerLimit           100
    MaxRequestsPerChild   1000
</IfModule>

I thought this would limit the number of httpd processes to 100?

From /var/log/messages:

May 17 08:54:47 ip-xxx-xxx-xxx-xxx kernel: [5955601.202712] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.206073] 359 total pagecache pages
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.207550] 0 pages in swap cache
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.208898] Swap cache stats: add 0, delete 0, find 0/0
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.211080] Free swap  = 0kB
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.212467] Total swap = 0kB
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.213700] 524189 pages RAM
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.214916] 0 pages HighMem/MovableOnly
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.216427] 11362 pages reserved
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.217763] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.221141] [ 1527]     0  1527     2865      230      11       3        0         -1000 udevd
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.224563] [ 2037]     0  2037     2340      125      10       3        0             0 dhclient
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.228241] [ 2120]     0  2120    62059     1234      24       3        0             0 rsyslogd
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.231735] [ 2149]    81  2149     5447       61      15       3        0             0 dbus-daemon
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.235328] [ 2289]     0  2289     5621       58      17       3        0             0 xinetd
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.238711] [ 2331]     0  2331    22246      463      45       3        0             0 sendmail
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.242348] [ 2340]    51  2340    20111      370      40       3        0             0 sendmail
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.245888] [ 2367]     0  2367    29888      157      16       4        0             0 crond
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.249419] [ 2381]     0  2381     4266       39      14       3        0             0 atd
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.252685] [ 2432]     0  2432     1614       29       8       3        0             0 agetty
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.256350] [ 2433]     0  2433     1077       22       8       3        0             0 mingetty
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.259756] [ 2436]     0  2436     1077       22       7       3        0             0 mingetty
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.263167] [ 2439]     0  2439     1077       23       8       3        0             0 mingetty
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.267901] [ 2441]     0  2441     1077       23       8       3        0             0 mingetty
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.271350] [ 2443]     0  2443     1077       22       8       3        0             0 mingetty
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.274733] [ 2445]     0  2445     1077       23       7       3        0             0 mingetty
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.278528] [18103]     0 18103   111419     2632     204       4        0             0 httpd
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.282315] [ 6601]     0  6601    19985      205      41       3        0         -1000 sshd
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.285707] [ 6633]     0  6633    13238      108      26       3        0         -1000 auditd
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.289151] [ 2642]    38  2642     7323      141      18       3        0             0 ntpd
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.292594] [ 6658]    48  6658   139739     6636     221       4        0             0 httpd
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.296161] [ 6662]    48  6662   138636     6634     220       4        0             0 httpd
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.299436] [ 6666]    48  6666   188897     6662     238       4        0             0 httpd
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.302676] [ 6670]    48  6670   139739     6725     221       4        0             0 httpd
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.306148] [ 6673]    48  6673   139730     6627     221       4        0             0 httpd
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.309594] [ 6678]    48  6678   138561     5454     219       4        0             0 httpd
May 17 08:54:49 ip-xxx-xxx-xxx-xxx kernel: [5955601.313091] [ 6701]    48  6701   139729     6629     221       4        0             0 httpd
.......

.......

Luke Wenke
  • 103
  • 4
  • "There were 140-160 processes of httpd running": the output you provided has NO evidence of this. How are you sure about the supposed-previously-running more-than-Maxclient apache processes? – Damiano Verzulli May 17 '17 at 05:40
  • The "From /var/log/messages" excerpt had 140-160 lines with "httpd" in it... I assume that was a list of the processes that were running at the time... this list appeared a few times when the OOM killer was running – Luke Wenke May 18 '17 at 00:37

1 Answers1

0

Error Cannot allocate memory: AH00159: fork: Unable to fork new process said that you don't have free Virtual memory on your server, more about virtual memory you could read in this answer. OOM killer do it work, when system couldn't use more Resident Memory then it exist in system, size of Resident Memory calculated as RAM+Swap. You don't have swap at all.

So easiest solution for you - add additional swap. How to do it mentioned in this article.

Alexander Tolkachev
  • 4,608
  • 3
  • 14
  • 23