0

I Have apache2 running on Ubuntu 22. My Apache2 often getting killed by OOM. I have tried to get log and its showing something like this

> root@localhost:~# dmesg -e | grep -i kill [Mar15 03:00] shopping_cart
> invoked oom-killer:
> gfp_mask=0x1100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0,
> oom_score_adj=0 [  +0.000004]  oom_kill_process.cold+0xb/0x10 [ 
> +0.000001] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/apache2.service,task=shopping_cart,pid=1676,uid=33
> [  +0.000012] Out of memory: Killed process 1676 (shopping_cart)
> total-vm:2398524kB, anon-rss:1834584kB, file-rss:0kB, shmem-rss:0kB,
> UID:33 pgtables:4044kB oom_score_adj:0 [  +4.614029] changelog.k2
> invoked oom-killer:
> gfp_mask=0x1100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0,
> oom_score_adj=0 [  +0.000003]  oom_kill_process.cold+0xb/0x10 [ 
> +0.000002] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/apache2.service,task=changelog.k2,pid=6244,uid=33 [  +0.000016] Out of memory: Killed process 6244 (changelog.k2)
> total-vm:2398524kB, anon-rss:1856988kB, file-rss:0kB, shmem-rss:0kB,
> UID:33 pgtables:4048kB oom_score_adj:0

I am not getting idea with above logs.

when I check status of apache2 when its get killed I am getting something like this

root@localhost:~# sudo systemctl status apache2
× apache2.service - The Apache HTTP Server
     Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
     Active: failed (Result: oom-kill) since Wed 2023-03-15 03:00:07 UTC; 9min ago
       Docs: https://httpd.apache.org/docs/2.4/
    Process: 620 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
    Process: 1746 ExecStop=/usr/sbin/apachectl graceful-stop (code=exited, status=0/SUCCESS)
   Main PID: 736 (code=exited, status=0/SUCCESS)
        CPU: 5min 4.064s

If I restart apache2, its start working fine but after some minutes, its again getting killed

I have tried apache2buddy and its showing me something like this

--------------------------------------------------------------------------------
### GENERAL FINDINGS & RECOMMENDATIONS ###
--------------------------------------------------------------------------------
Apache2buddy.pl report for server: localhost (45.33.10.196):

Settings considered for this report:
[ !! ] *** LOW UPTIME ***.
[ @@ ] The following recommendations may be misleading - apache has been restarted within the last 24 hours.

        Your server's physical RAM:                                   3923 MB
        Remaining Memory after other services considered:             3923 MB
        Apache's MaxRequestWorkers directive:                         115      <--------- Current Setting
        Apache MPM Model:                                             prefork
        Largest Apache process (by memory):                           40 MB
[ !! ]  Your MaxRequestWorkers setting is too high.
        Your recommended MaxRequestWorkers setting (based on available memory) is between 87 and 97. <-- Acceptable Range (90-100% of Remaining RAM)
        Max potential memory usage:                                   4646 MB
        Percentage of TOTAL RAM allocated to Apache:                  118.43  %
        Percentage of REMAINING RAM allocated to Apache:              118.43  %
--------------------------------------------------------------------------------

I have only changed mode_prefork like this

<IfModule mpm_prefork_module>
        StartServers              2
        MinSpareServers           5
        MaxSpareServers          10
        MaxRequestWorkers        115
        MaxConnectionsPerChild   0
</IfModule>

I think its getting because of memory but I am not able to figure out which process of apache is consuming much memory and how I can solve it. I have tried htop and free but its not giving me any clue.

Let me know if someone here can help me for solve the puzzle.

Thanks!

  • We need to see your Apache config, obviously. Don't dump the whole config, only the parts you were added/changed. Also, try to use `mod_status` to know what requests it was processing when killed, and then trace requests to whatever program they invoke. – Nikita Kipriyanov Mar 15 '23 at 05:44
  • @NikitaKipriyanov I have added pre_fork code. Thanks! –  Mar 15 '23 at 06:23
  • Reduce `MaxRequestWorkers` by half, wait for a day to make sure that you are not getting OOMs. Start increasing the number daily until you start getting OOMs again. Reduce to previous value. – AlexD Mar 15 '23 at 06:30
  • 1
    apache2buddy already told you what the problem is. But you should also read it. "Apache's MaxRequestWorkers directive: 115 <--------- Current Setting", "Your recommended MaxRequestWorkers setting (based on available memory) is between 87 and 97. <-- Acceptable Range (90-100% of Remaining RAM)", "Percentage of TOTAL RAM allocated to Apache: 118.43 %"... Either reduce MaxRequestWorkers, increase RAM or add some SWAP (not recommend). – paladin Mar 15 '23 at 07:43

0 Answers0