3

Apache on my webserver is suddenly very slow. I have a 8 GB Ram, 4 CPU server at digitalocean with LAMP installed. What can I do to fix this?

This is the top output:

top - 10:37:15 up 93 days, 20:20,  1 user,  load average: 0.22, 0.22, 0.28
Tasks: 471 total,   3 running, 468 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.4 us,  0.6 sy,  0.0 ni, 97.4 id,  0.1 wa,  0.0 hi,  0.5 si,  0.0 st
KiB Mem:   8176832 total,  6816036 used,  1360796 free,   184160 buffers
KiB Swap:        0 total,        0 used,        0 free.  5057196 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                
 6463 mysql     20   0 2420696  88740   3808 S   0.7  1.1 348:23.30 mysqld                                                                                                 
16207 www-data  20   0  385076  11560   4376 S   0.7  0.1   0:00.18 apache2                                                                                                
    7 root      20   0       0      0      0 S   0.3  0.0 818:55.76 rcu_sched                                                                                              
    8 root      20   0       0      0      0 S   0.3  0.0 464:04.83 rcuos/0                                                                                                
 1053 root      20   0  100708   2624    708 S   0.3  0.0   9:19.16 sendmail-mta                                                                                           
16168 www-data  20   0  384812  11420   4424 S   0.3  0.1   0:00.17 apache2                                                                                                
16194 www-data  20   0  385076  11620   4384 S   0.3  0.1   0:00.21 apache2                                                                                                
16202 www-data  20   0  384820  11432   4428 S   0.3  0.1   0:00.22 apache2                                                                                                
16276 www-data  20   0  385076  11588   4440 S   0.3  0.1   0:00.15 apache2                                                                                                
16287 www-data  20   0  386696  13136   4436 S   0.3  0.2   0:00.18 apache2                                                                                                
16309 www-data  20   0  385068  11592   4404 S   0.3  0.1   0:00.17 apache2                                                                                                
16333 www-data  20   0  384812  11392   4396 S   0.3  0.1   0:00.14 apache2                                                                                                
16340 www-data  20   0  386348  12608   4416 R   0.3  0.2   0:00.11 apache2                                                                                                
16353 www-data  20   0  386696  13036   4404 S   0.3  0.2   0:00.22 apache2                                                                                                
16385 www-data  20   0  385076  11640   4404 S   0.3  0.1   0:00.21 apache2                                                                                                
16412 www-data  20   0  386696  13116   4436 S   0.3  0.2   0:00.40 apache2                                                                                                
16446 www-data  20   0  386672  13080   4404 S   0.3  0.2   0:00.14 apache2                                                                                                
16454 www-data  20   0  386696  13100   4424 S   0.3  0.2   0:00.17 apache2                                                                                                
16479 www-data  20   0  386412  12852   4396 S   0.3  0.2   0:00.14 apache2                                                                                                
16540 root      20   0   25212   1988   1176 R   0.3  0.0   0:00.33 top                                                                                                    
    1 root      20   0   33500   2688   1328 S   0.0  0.0   0:44.63 init                                                                                                   
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.15 kthreadd                                                                                               
    3 root      20   0       0      0      0 S   0.0  0.0 202:28.30 ksoftirqd/0                                                                                            
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H                                                                                           
    6 root      20   0       0      0      0 S   0.0  0.0 157:44.10 kworker/u8:0                                                                                           
    9 root      20   0       0      0      0 R   0.0  0.0 164:29.67 rcuos/1                                                                                                
   10 root      20   0       0      0      0 S   0.0  0.0 152:37.93 rcuos/2                                                                                                
   11 root      20   0       0      0      0 S   0.0  0.0  68:24.97 rcuos/3                                                                                                
   12 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh                                                                                                 
   13 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/0                                                                                                
   14 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/1                                                                                                
   15 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/2

This is the apache error log:

[Tue Dec 29 10:32:11.628945 2015] [:error] [pid 16404] [client 61.178.71.8:37626] PHP Notice:  Undefined variable: config_location in /var/www/html/a/lib/common.php on line 32, referer: http://hi-techno.xyz/clicknety.html
[Tue Dec 29 10:32:11.727455 2015] [:error] [pid 16447] [client 87.168.254.145:63259] PHP Notice:  Undefined variable: countryname in /var/www/html/c/arth.php(33) : eval()'d code on line 1, referer: http://eywow.com/c/arth.php?slug=balls
[Tue Dec 29 10:32:11.727566 2015] [:error] [pid 16447] [client 87.168.254.145:63259] PHP Notice:  Undefined variable: city in /var/www/html/c/arth.php(33) : eval()'d code on line 1, referer: http://eywow.com/c/arth.php?slug=balls
[Tue Dec 29 10:32:11.728356 2015] [:error] [pid 16447] [client 87.168.254.145:63259] PHP Notice:  Undefined offset: -1 in /var/www/html/c/arth.php on line 323, referer: http://eywow.com/c/arth.php?slug=balls
[Tue Dec 29 10:32:12.077442 2015] [:error] [pid 16326] [client 138.0.68.53:2349] PHP Notice:  A session had already been started - ignoring session_start() in /var/www/html/a/lib/common.php on line 3, referer: http://hi-techno.xyz/clicknety.html
[Tue Dec 29 10:32:12.077514 2015] [:error] [pid 16326] [client 138.0.68.53:2349] PHP Notice:  Undefined index: domain in /var/www/html/a/lib/common.php on line 14, referer: http://hi-techno.xyz/clicknety.html
[Tue Dec 29 10:32:12.077950 2015] [:error] [pid 16326] [client 138.0.68.53:2349] PHP Notice:  Undefined variable: config_location in /var/www/html/a/lib/common.php on line 32, referer: http://hi-techno.xyz/clicknety.html
[Tue Dec 29 10:32:12.213301 2015] [:error] [pid 16220] [client 187.111.210.17:4461] PHP Notice:  A session had already been started - ignoring session_start() in /var/www/html/a/lib/common.php on line 3, referer: http://hi-techno.xyz/clicknety.html
[Tue Dec 29 10:32:12.213353 2015] [:error] [pid 16220] [client 187.111.210.17:4461] PHP Notice:  Undefined index: domain in /var/www/html/a/lib/common.php on line 14, referer: http://hi-techno.xyz/clicknety.html
[Tue Dec 29 10:32:12.213812 2015] [:error] [pid 16220] [client 187.111.210.17:4461] PHP Notice:  Undefined variable: config_location in /var/www/html/a/lib/common.php on line 32, referer: http://hi-techno.xyz/clicknety.html
[Tue Dec 29 10:32:12.902652 2015] [:error] [pid 16256] [client 162.156.92.161:57060] PHP Notice:  Undefined variable: countryname in /var/www/html/c/arth.php(33) : eval()'d code on line 1, referer: http://eywow.com/c/arth.php?slug=ufo
[Tue Dec 29 10:32:12.902836 2015] [:error] [pid 16256] [client 162.156.92.161:57060] PHP Notice:  Undefined variable: city in /var/www/html/c/arth.php(33) : eval()'d code on line 1, referer: http://eywow.com/c/arth.php?slug=ufo
[Tue Dec 29 10:32:13.002648 2015] [:error] [pid 16427] [client 41.190.198.114:50583] PHP Notice:  A session had already been started - ignoring session_start() in /var/www/html/a/lib/common.php on line 3, referer: http://hi-techno.xyz/clicknety.html
[Tue Dec 29 10:32:13.002704 2015] [:error] [pid 16427] [client 41.190.198.114:50583] PHP Notice:  Undefined index: domain in /var/www/html/a/lib/common.php on line 14, referer: http://hi-techno.xyz/clicknety.html
[Tue Dec 29 10:32:13.003456 2015] [:error] [pid 16427] [client 41.190.198.114:50583] PHP Notice:  Undefined variable: config_location in /var/www/html/a/lib/common.php on line 32, referer: http://hi-techno.xyz/clicknety.html
[Tue Dec 29 10:32:13.136681 2015] [:error] [pid 16256] [client 119.93.37.113:38315] PHP Notice:  A session had already been started - ignoring session_start() in /var/www/html/a/lib/common.php on line 3, referer: http://hi-techno.xyz/clicknety.html
[Tue Dec 29 10:32:13.136740 2015] [:error] [pid 16256] [client 119.93.37.113:38315] PHP Notice:  Undefined index: domain in /var/www/html/a/lib/common.php on line 14, referer: http://hi-techno.xyz/clicknety.html
[Tue Dec 29 10:32:13.137251 2015] [:error] [pid 16256] [client 119.93.37.113:38315] PHP Notice:  Undefined variable: config_location in /var/www/html/a/lib/common.php on line 32, referer: http://hi-techno.xyz/clicknety.html

My site example page is very slow, this is the server status page of apache My guess is that the iptables are slowing it down, as there is no reason from the hardware side, there are enough free resources. I have 8 GB Ram and 4 CPUs.

Michael Hampton
  • 244,070
  • 43
  • 506
  • 972
user77431
  • 33
  • 1
  • 4
  • Lots of PHP errors in that log you should be fixing. No Apache errors though. – ceejayoz Dec 29 '15 at 15:50
  • "I know it is only Apache because my html only frontpage loads very fast (under 1 second)." That means it's **not** Apache. Apache serves that HTML - if static HTML is fast, Apache isn't the issue. Your problem likely lies in your code and/or any databases it's hooking into. – ceejayoz Dec 29 '15 at 15:50
  • @ceejayoz Sorry for that but seems like it was just cached, when I load it on my other pc it is slow, too. Can the PHP errors be the cause of the problems? They were there since ever and it loaded very fast before. – user77431 Dec 29 '15 at 15:53
  • We have little to go on. Your server's load is small - 0.22. You have free RAM. You have a variety of PHP errors you should fix, but chances are the issue lies elsewhere. We can't really debug this remotely - you need to benchmark with your browser's inspector and some simple scripts and see where the bottleneck is. – ceejayoz Dec 29 '15 at 15:56
  • @ceejayoz It runs on ubuntu, with what tools/tests should I start? – user77431 Dec 29 '15 at 16:07
  • I guarantee you that iptables is not slowing it down. – Wesley Dec 29 '15 at 23:54
  • You posted an example URL (http://eywow.com/a/test.php) in the other post, and it loads quickly for me - 1 MB in ~2 seconds. – ceejayoz Dec 30 '15 at 00:29
  • @ceejayoz Thanks for your time, but the error got fixed about 30 mins ago. Increasing the MaxClients solved it. – user77431 Dec 30 '15 at 00:37

2 Answers2

1
63.2 requests/sec - 1.4 MB/second - 23.4 kB/request
375 requests currently being processed, 0 idle workers

You've maxed out your available Apache processes. You need to increase maxclients. Check your Apache error logs for more verbose information about the problem.

Wesley
  • 32,690
  • 9
  • 82
  • 117
  • I already did the math today, one apache process needs about 12 MB, so 8GB - 1 GB for buffering, 7 GB/12 MB = about 600. I just canged it to StartServers 100 MinSpareServers 5 MaxSpareServers 200 ServerLimit 600 MaxClients 600 MaxRequestsPerChild 0 – user77431 Dec 30 '15 at 00:03
  • @MichaelHampton load average is low, CPU has plenty of headroom, but there's a relatively high amount of requests per second. Seems like it's just more clients than processes and new requests are waiting for a free one even if the request takes just a second or two to fulfill. Who knows... I do not know computer well. ¯\\(°_°)/¯ – Wesley Dec 30 '15 at 00:04
  • @MichaelHampton So I just need to upgrade my sever? – user77431 Dec 30 '15 at 00:05
  • You're right, something else is going on here. – Michael Hampton Dec 30 '15 at 00:06
  • Well after this change it is working fast like before, the server handles about 900 visitors at once and 30Mbit/s average, but sometimes this "slow down" happens. I changed nothing in the last days or weeks. – user77431 Dec 30 '15 at 00:08
  • @Wesley This fixed it for the moment, thanks. Do you think I should upgrade my server to prevent this errors in the future? – user77431 Dec 30 '15 at 00:16
  • @user77431 that's a decent start but if you continue to grow you might need to consider other measures. Once you get to 1500 to 2000 simultaneous visitors you might need to toss a caching proxy into the stack or load balance. – Wesley Dec 30 '15 at 00:31
  • @Wesley I will upgrade it to 2x times its size `16 GB` and `8 Cores`, can I just change ServerLimit 600 MaxClients 600 to ServerLimit 1200 MaxClients 1200 and thats it? Or should I also adjust other settings with a bigger server? – user77431 Dec 30 '15 at 00:46
  • @user77431 There's other considerations, but generally speaking yes, just double the MacClients for the quick solution. I think you may be hitting capacity limits with the current settings though. You should be able to handle more than just 1200 concurrent sessions with that hardware, unless there's also a database server and other applications running on that host. Contact me with info in my profile if you'd like some consulting. – Wesley Dec 30 '15 at 00:48
0

Can you post the results of the following?

top -u

This should give you an overview of all processes on your server ranking by CPU usage. You may also want to grep out everything leaving only Apache. 

top -u | grep Apache

There could be some other resource hogs or many a memory leak somewhere down the line.

Felix Jen
  • 403
  • 5
  • 18