4

On an Ubuntu server with 1Gb of RAM, an Apache 2.4.7 server with default configuration started to occasionally stop responding to request. It's used for a personal cloud + other needs and a low traffic website.

An error.log examination revealed this pattern that seem recurrent each time the problem appears:

[mpm_prefork:error] [pid 31950] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

after which access.log doesn't log any access at all. Then a day later:

[mpm_prefork:notice] [pid 31950] AH00171: Graceful restart requested, doing restart

which apparently didn't make any difference.

I've had to restart the server manually each time.

Now I'm looking at mpm_prefork.conf, which is default and from the info I've been able to gather, I don't think I need to change anything. I've started suspecting there could be some memory leak and read that MaxConnectionsPerChild 1000 is kind of a hack to prevent memory leak, so I put it to this 1000 value and will see how it behaves. Since it's once in a while error, it's difficult to understand what's causing the problem exactly.

Do you think using MaxConnectionsPerChild in this context is a good strategy (hoping the memory leak will be fixed in further release of the site I'm hosting), or does this just not indicate a memory leak?

NB: Having an average Apache process consuming ~20Mb of memory, I reduced the MaxRequestWorkers from 150 to 30 (considering 500Mb of mem available for Apache on the server). I have now:

<IfModule mpm_prefork_module>
        StartServers              5
        MinSpareServers           5
        MaxSpareServers           10
        MaxRequestWorkers         30
        MaxConnectionsPerChild    1000
</IfModule>

I don't think the memory peaked out, but as my Munin client was down, I realize now, I can't exclude it.

coolnodje
  • 233
  • 1
  • 3
  • 8
  • Same problem occurred this morning again, despite the changes: `[mpm_prefork:error] [pid xxxxx] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers`. ~30 Apache processes, 600Mb MemFree. Apache doesn't answer, but telnet on 80 connects. Can't do anything but restart. Installed Monit to have possibly more infos... – coolnodje Feb 17 '15 at 02:53
  • This is in fact happening every few hours... There's no error in syslog, and no more information in Apache's error.log. I'm out of ideas on how to track the problem, input welcome. – coolnodje Feb 18 '15 at 10:50

3 Answers3

2

Removing Owncloud 8.0.0 from my document root directory is the ultimate answer. After that Apache just works as expected...

It would have been good to understand what was causing the problem though and how to configure Apache so that it can handle problematic PHP application. It's disappointing that MaxConnectionsPerChild is not able to handle the problem in this case, but again, I'm not sure what happened really, even though the MaxRequestWorkers error log seemed to indicate the number of thread was the issue.

coolnodje
  • 233
  • 1
  • 3
  • 8
1

Maybe my situation is too unique, I was getting the same server reached MaxRequestWorkers setting error and believed that mine was also a low traffic site. Otherwise the logs were barren. I found a post that said I could up Apache’s logging level from ‘warn’ to ‘debug’, so I changed it hoping to glean something useful. I was tailing the /var/log/apache2/error.log when I restarted Apache, and the output immediately exploded into an endless scrolling mess. I could not read it as it was going by so fast. I did notice that every 1000 lines or so there was normal line, but the rest of the lines were identical – all originating from a Russian IP. My first DOS!

I read that you could drop packets from a specific ip using a line in iptables. I ran two lines and the scrolling stopped – COLD. 

iptables -A INPUT -s <ip address here> -j DROP
iptables -A INPUT -s <ip address here> -j DROP

There are many good resources about how to mitigate DOS attacks, but for me, I did not yet know that I was under one. Following this [mpm_prefork:error] clue was my big break. I am sure that not every prefork error is a DOS attack, but maybe this post can help someone!

https://linuxaria.com/howto/how-to-verify-ddos-attack-with-netstat-command-on-linux-terminal

https://stackoverflow.com/questions/38357269/mpm-preforkerror-ah00161-server-reached-maxrequestworkers-setting/38362719

iptables rules to counter the most common DoS attacks?

Community
  • 1
wruckie
  • 678
  • 6
  • 22
0

Do you perhaps have a bunch of processes sitting waiting for subrequests to apache, while those processes ensure apache never has a spare process to answer with?

It's important that any call to an url on the same server has a timeout.

mc0e
  • 5,866
  • 18
  • 31