2

I have an openSUSE 13.1 VM (host runs Virtualbox 4.2.18, also on openSUSE 13.1) and restarting httpd (Apache/2.4.6) always takes 1.5 minute:

foobar:~ # time /etc/init.d/apache2 restart
redirecting to systemctl restart apache2.service

real    1m30.778s
user    0m0.004s
sys     0m0.000s

Immediately subsequent restart is normal (very fast):

foobar:~ # time /etc/init.d/apache2 restart
redirecting to systemctl restart apache2.service

real    0m1.023s
user    0m0.004s
sys     0m0.000s

5 minutes later the restart time goes again to exactly 90 seconds:

foobar:/tmp # time /etc/init.d/apache2 restart
redirecting to systemctl restart apache2.service

real    1m30.684s
user    0m0.000s
sys     0m0.000s

What I've looked for so far:

  • top while apache is restarting doesn't show a lot (~0% usage).
  • netstat also doesn't show any connections with the outside world.

Note that this is a VM which currently has 0 traffic and there are plenty of free GBs available in memory and disk.

I've also found that it's the "stop" part of the "restart" is what takes 90 seconds.

Any idea why this is happening or where should I look at next?

Edit: I found out that when stop takes 90 seconds I consistently get the following in /var/log/apache2/error_log:

[core:notice] [pid 3179] AH00052: child pid 3203 exit signal Segmentation fault (11)
cherouvim
  • 794
  • 3
  • 21
  • 37
  • Two things: 1. top shows not only per process usage, but what the cpu currently does (3rd line from the top). So check exactly what your cpu is doing - maybe it is waiting for something. 2. I just read that starting a process nowadays consumes entropy. VMs are notorious for having low entropy. Monitor your available entropy by repeatedly looking at `cat /proc/sys/kernel/random/entropy_avail`. If it is in the low hundreds, you may need to increase entropy, e.g. using haveged. – Isaac Feb 11 '14 at 07:51
  • Likely workers aren't exiting when asked and are getting killed only after the grace period expires. – Brian Feb 11 '14 at 09:29
  • @lsaac: 3rd line shows nothing. Idle. I've never heard about entropy before but `cat` gives me around 150. Also, I've found that apache is slow only when stopping. – cherouvim Feb 11 '14 at 10:10

3 Answers3

1

When using graceful restart, the parent apache process stops accepting new connections and waits forever for all the child processes to exit. So essentially the web server is dead (other than existing connections) until all the existing children exit.

In the normal use case of short-lived http/https connections, this is not a problem when doing a graceful shutdown or restart...it should normally take a second. The problem is when you have something that delays the children exiting, such as persistent websocket connections. In that case the server will never actually manage to gracefully stop/restart...it will just sit there forever in a semi-dead state.

You can adjust the delay using the GracefulShutdownTimeout directive:

https://httpd.apache.org/docs/2.4/mod/mpm_common.html#gracefulshutdowntimeout

By default it is set to 0 (infinite). 5 seconds is a more reasonable value.

Note that when using systemctl to restart the server, it will only wait a maximum of 90 seconds by default before forcing it to kill the child processes (rather than forever), which is why you are seeing this 90 second delay. This is set in /etc/systemd/system.conf:

#DefaultTimeoutStopSec=90s

This can also be changed for individual units using the TimeoutStopSec option.

CpnCrunch
  • 121
  • 3
1

After a lot of trial and error I found out that it's being caused by the php5 module which is being loaded in /etc/sysconfig/apache2. Removing it completelly stopped this behavior.

I need the php5 module though, so in order to mitigate this delay I've added the following in etc/apache2/server-tuning.conf:

GracefulShutdownTimeout 2

Now when the segfault occurs when stopping apache it only hangs for 2 seconds.

cherouvim
  • 794
  • 3
  • 21
  • 37
1

This same thing was happening to me and it turns out that I left the /etc/hosts file at its default setting.

The delay went away immediately after I updated the hosts file similar to this:

127.0.0.1   localhost
::1         localhost
172.16.333.444  www.mysite.com mysite.com

References:

/etc/hosts entry for single IP server serving multiple domains

https://unix.stackexchange.com/questions/57439/slow-start-of-midnight-commander#answer-397879

cssyphus
  • 348
  • 2
  • 8