0

I sometimes notice troubles on my computer about HTTP server. Indeed, no more incoming request is accepted by the service. It just replies by an error code 503.

Command:

apachectl fullstatus

gives

                  Apache Server Status for localhost (via ::1)

   Server Version: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_jk/1.2.43

   Server MPM: worker

   Server Built: Apr 20 2018 18:10:38

     ----------------------------------------------------------------------

   Current Time: Wednesday, 09-Oct-2019 06:00:21 UTC

   Restart Time: Monday, 07-Oct-2019 06:58:15 UTC

   Parent Server Config. Generation: 1

   Parent Server MPM Generation: 0

   Server uptime: 1 day 23 hours 2 minutes 6 seconds

   Server load: 0.27 0.15 0.30

   Total accesses: 10981607 - Total Traffic: 10.6 GB

   CPU Usage: u3.9 s3.53 cu0 cs0 - .00439% CPU load

   64.9 requests/sec - 65.6 kB/second - 1035 B/request

   2500 requests currently being processed, 0 idle workers

 WWCWWCCWRCWCCCWRCCCRWCCCCCCCWCWCCCCCCCCCCCCCCCCCCCCCCCCWRCCCCCCC
 CCCCCWCRWCWWCWWCWWWCCCCCCCCCCCCCCCCCWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWRWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
 WWWW

   Scoreboard Key:
   "_" Waiting for Connection, "S" Starting up, "R" Reading Request,
   "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
   "C" Closing connection, "L" Logging, "G" Gracefully finishing,
   "I" Idle cleanup of worker, "." Open slot with no current process

Srv  PID   Acc        M CPU  SS   Req    Conn Child Slot  Client          VHost              Request                                                     
0-0  11751 0/1/12656  W 0.02 86   0      0.0  0.00  10.55 166.253.230.136 217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 0/0/12595  W 0.22 131  0      0.0  0.00  15.87 80.187.116.81   217.12.194.115:80 POST                                                        
                                                                                             /
0-0  11751 1/23/12655 C 0.12 0    0      0.8  0.02  13.38 63.41.170.251   217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 0/3/12396  W 0.02 66   0      0.0  0.00  10.18 80.187.116.81   217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 0/3/12712  W 0.02 68   0      0.0  0.00  15.82 80.187.99.192   217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/13/12640 C 0.10 0    0      0.8  0.01  10.26 27.253.251.141  217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/4/12592  C 0.12 0    0      0.9  0.00  10.42 213.225.5.252   217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 0/0/12775  W 0.06 131  0      0.0  0.00  11.11 109.41.128.180  217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 0/14/12875 R 0.09 0    0      0.0  0.01  15.28 182.232.146.125 
0-0  11751 1/15/12655 C 0.11 0    0      0.8  0.01  13.54 130.226.56.9    217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 0/0/12409  W 0.22 46   0      0.0  0.00  11.48 223.38.61.134   217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/12/12337 C 0.12 0    0      0.8  0.01  11.08 166.218.54.143  217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/6/12526  C 0.10 0    0      0.8  0.01  10.72 202.214.198.41  217.12.194.115:80 POST                                                        
                                                                                             /
0-0  11751 1/13/12245 C 0.11 0    0      0.9  0.01  9.87  195.47.211.230  217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 0/0/12561  W 0.22 77   0      0.0  0.00  10.69 109.38.138.199  217.12.194.115:80 POST                                                        
                                                                                             /
0-0  11751 0/3/12993  R 0.05 1    0      0.0  0.00  21.49 185.184.6.153   
0-0  11751 1/14/12333 C 0.12 0    0      0.9  0.01  14.12 223.38.61.10    217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/5/12287  C 0.09 0    0      0.9  0.00  10.38 80.187.110.210  217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/13/12353 C 0.12 0    0      0.9  0.01  10.41 107.241.129.44  217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 0/11/12425 R 0.04 0    0      0.0  0.01  17.22 182.232.146.125 
0-0  11751 0/0/12290  W 0.07 126  0      0.0  0.00  10.36 27.253.251.246  217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/11/12315 C 0.11 0    0      0.9  0.01  10.23 80.187.119.118  217.12.194.115:80 POST                                                        
                                                                                             /
0-0  11751 1/9/12321  C 0.11 0    0      0.8  0.01  10.02 202.214.125.247 217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/11/12501 C 0.11 0    0      0.8  0.01  10.69 143.97.99.17    217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/10/12415 C 0.10 0    0      0.9  0.01  10.74 223.39.188.243  217.12.194.115:80 POST                                                        
                                                                                             /
0-0  11751 1/6/12273  C 0.06 1    0      0.8  0.01  10.67 80.187.120.70   217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/23/12211 C 0.12 0    0      1.0  0.02  10.14 109.40.131.188  217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/19/12735 C 0.11 0    0      1.0  0.02  11.18 37.81.149.127   217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 0/16/12325 W 0.10 0    0      0.0  0.01  11.61 ::1             217.12.194.115:80 GET /server-status HTTP/1.1                                 
0-0  11751 1/2/12409  C 0.10 0    0      0.9  0.00  11.34 223.38.62.142   217.12.194.115:80 POST                                                        
                                                                                             /
0-0  11751 0/1/12414  W 0.02 61   0      0.0  0.00  10.75 223.38.60.237   217.12.194.115:80 POST                                                        
                                                                                             /
0-0  11751 1/12/12250 C 0.11 0    0      0.9  0.01  10.24 166.167.75.65   217.12.194.115:80 POST                                                        
                                                                                             /
0-0  11751 1/8/12153  C 0.09 0    0      0.8  0.01  10.66 210.149.254.99  217.12.194.115:80 POST                                                        
                                                                                             /
0-0  11751 1/11/12392 C 0.11 0    0      0.9  0.01  10.47 192.38.140.206  217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/11/12717 C 0.10 0    0      0.9  0.01  12.26 107.241.129.61  217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/30/12380 C 0.12 0    0      0.9  0.03  11.24 223.39.188.39   217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/14/12641 C 0.10 0    0      0.9  0.01  11.67 159.242.15.20   217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/14/12570 C 0.11 0    0      0.8  0.01  14.79 1.125.110.139   217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/10/12304 C 0.10 0    0      0.8  0.01  11.63 186.11.160.77   217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/4/12110  C 0.04 2    0      0.8  0.00  12.95 49.111.12.14    217.12.194.115:80 POST /
                                                                                             HTTP/1                                                      
0-0  11751 1/6/12239  C 0.12 0    0      0.9  0.01  13.58 46.96.26.208    217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/16/12451 C 0.12 0    0      0.9  0.01  14.74 80.187.82.107   217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/6/12354  C 0.10 0    0      0.9  0.01  10.60 122.56.198.205  217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/17/12055 C 0.12 0    0      0.9  0.01  12.80 130.226.56.9    217.12.194.115:80 POST                                                        
                                                                                             /
0-0  11751 1/17/12121 C 0.11 0    0      0.9  0.02  13.04 213.225.37.2    217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/26/12466 C 0.12 0    0      0.8  0.02  14.10 107.241.129.67  217.12.194.115:80 POST                                                        
                                                                                             http://
0-0  11751 1/14/12128 C 0.09 0    0      0.9  0.01  12.49 109.166.139.108 217.12.194.115:80 POST                                                        
                                                                                             /
0-0  11751 1/9/12274  C 0.12 0    0      0.9  0.01  13.05 80.187.114.172  217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/17/12243 C 0.12 0    0      0.8  0.01  10.42 166.253.230.136 217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/18/12382 C 0.12 0    0      0.8  0.02  13.73 210.149.255.114 217.12.194.115:80 POST                                                        
                                                                                             /
0-0  11751 1/22/12059 C 0.10 0    0      0.9  0.02  10.11 107.241.129.66  217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/6/12235  C 0.12 0    0      0.9  0.01  10.66 80.187.108.103  217.12.194.115:80 POST                                                        
                                                                                             /
0-0  11751 1/11/12337 C 0.12 0    0      0.9  0.01  11.05 80.187.83.0     217.12.194.115:80 POST                                                        
                                                                                             /
0-0  11751 1/14/12178 C 0.09 0    0      0.8  0.01  11.41 210.138.179.250 217.12.194.115:80 POST                                                        
                                                                                             /
0-0  11751 1/22/12349 C 0.12 0    0      0.9  0.02  12.56 146.0.216.54    217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 0/0/12088  W 0.22 116  0      0.0  0.00  10.53 210.138.178.34  217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 0/6/12310  R 0.05 1    0      0.0  0.00  10.72 107.241.129.23  
0-0  11751 1/4/12289  C 0.12 0    0      0.8  0.00  10.21 210.138.179.250 217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/5/12286  C 0.12 0    0      1.1  0.00  13.53 80.187.98.229   217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/5/12336  C 0.12 0    0      1.0  0.00  13.34 107.77.92.131   217.12.194.115:80 POST                                                        
                                                                                             /
0-0  11751 1/2/12210  C 0.11 0    0      0.9  0.00  15.85 63.41.20.6      217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/3/12235  C 0.11 0    0      0.8  0.00  15.97 166.130.94.203  217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/1/11867  C 0.10 0    131168 0.9  0.00  9.79  80.187.110.210  217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/1/12377  C 0.10 0    131168 0.9  0.00  15.89 223.39.188.18   217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/2/11974  C 0.10 0    0      0.8  0.00  10.13 210.149.255.149 217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/1/12017  C 0.10 0    131162 0.9  0.00  13.18 223.38.60.237   217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/3/12393  C 0.11 0    0      0.9  0.00  12.91 223.38.63.98    217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/5/12346  C 0.12 0    0      0.8  0.00  10.81 210.138.178.34  217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/2/12156  C 0.12 0    0      0.9  0.00  16.26 80.187.108.51   217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 0/0/11894  W 0.22 116  0      0.0  0.00  11.10 109.41.64.164   217.12.194.115:80 POST                                                        
                                                                                             /
0-0  11751 1/3/12170  C 0.12 0    0      1.1  0.00  16.21 42.106.7.0      217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 0/2/12469  R 0.10 0    0      0.0  0.00  21.02 192.38.138.119  
0-0  11751 0/1/12102  W 0.02 22   0      0.0  0.00  13.02 80.146.209.106  217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 1/1/12473  C 0.10 0    131124 0.9  0.00  10.63 223.39.191.248  217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 0/1/12149  W 0.02 89   0      0.0  0.00  13.94 80.75.32.22     217.12.194.115:80 GET                                                         
                                                                                             /
0-0  11751 0/1/12225  W 0.02 72   0      0.0  0.00  10.46 80.12.33.121    217.12.194.115:80 POST                                                        
                                                                                             /
1-0  3993  0/0/11961  W 0.34 511  0      0.0  0.00  10.68 93.23.104.148   217.12.194.115:80 GET                                                         
                                                                                             /
1-0  12187 0/0/11883  W 0.23 80   0      0.0  0.00  10.28 107.241.129.44  217.12.194.115:80 GET                                                         
                                                                                             /
1-0  12187 0/0/12023  W 0.23 80   0      0.0  0.00  16.24 27.253.251.202  217.12.194.115:80 GET                                                         
                                                                                             /
1-0  12187 0/0/12249  W 0.23 80   0      0.0  0.00  10.84 213.225.10.205  217.12.194.115:80 POST                                                        
                                                                                             /
1-0  12187 0/0/11666  W 0.23 80   0      0.0  0.00  9.95  185.184.6.153   217.12.194.115:80 GET                                                         
                                                                                             /
1-0  12187 0/0/11855  W 0.22 80   0      0.0  0.00  9.82  27.253.251.129  217.12.194.115:80 POST                                                        

     ----------------------------------------------------------------------

   Srv   Child Server number - generation                            
   PID   OS process ID                                               
   Acc   Number of accesses this connection / this child / this slot 
   M     Mode of operation                                           
   CPU   CPU usage, number of seconds                                
   SS    Seconds since beginning of most recent request              
   Req   Milliseconds required to process most recent request        
   Conn  Kilobytes transferred this connection                       
   Child Megabytes transferred this child                            
   Slot  Total megabytes transferred this slot                       

     ----------------------------------------------------------------------

   SSL/TLS Session Cache Status:                                      
   cache type: SHMCB, shared memory: 512000 bytes, current entries: 0 
   subcaches: 32, indexes per subcache: 88                            
   index usage: 0%, cache usage: 0%                                   
   total entries stored since starting: 598                           
   total entries replaced since starting: 0                           
   total entries expired since starting: 590                          
   total (pre-expiry) entries scrolled out of the cache: 0            
   total retrieves since starting: 452 hit, 3405 miss                 
   total removes since starting: 0 hit, 0 miss

MPM worker configuration:

<IfModule mpm_worker_module>
        StartServers        2
        MinSpareThreads     25
        MaxSpareThreads     2500
        ThreadLimit     100
        ThreadsPerChild     100
        ServerLimit     25
        MaxRequestWorkers   2500
        MaxConnectionsPerChild  1000
</IfModule>

Two recurrent errors are logged each time (/var/log/httpd/error_log) that warn me a problem is coming. It is a result of sudden increasing of http connexions and threads. I've not detected any sign of cyber attack.

  1. AH00286: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
  2. AH00287: server is within MinSpareThreads of MaxRequestWorkers, consider raising the MaxRequestWorkers setting

Have you an idea about what is going on ? On my server there are a lot of system (~200) that communicate with him every 5 seconds by using long polling request.

Note:

  • centOS 7
  • httpd 2.4.6
  • tomcat 7.0.76

UDPATE

I did not see any errors/warnings in tomcat logs, neigher in spring servlets logs. I set KeepAliveTimeout option (from HTTPD configuration) to 5 seconds but I seems to be ineffective (effective only on status '.' and '_' not 'W').

Change design of application by using new connexions instead of long polling requests is impossible, I guess, for my use case (reverse server-client req-res protocol).

In regular situation, total connection number is stable (~2000 including all states CLOSE_WAIT, CLOSING, CONNECTED, ...) between server and remotes systems, but for ununderstanding reasons this number exploses, indeed, this number reaches 15000 HTTP connextions in only 200 seconds equitably distributed on all remotes systems.

double_g
  • 101
  • 3
  • way too many C's and W's, you seem to have connection issues where the receiving end does not want to close connections. Also, maxconnectionsperchild is awefully low, set it to millions, if you can use a native mod_proxy_ajp, even better. – Daniel Ferradal Nov 29 '19 at 17:47

1 Answers1

1

Is it possible that those long polling requests (or whatever is generating those requests against your server) are not closing their connections properly after their work is finished?

Or more likely to open new connections instead implementing long-polling properly?

As theoretically if your long polling works as it should there should be no more than 200 connections open at any one time as all your remote systems keep the channel open and await the server to get back with new data right?

If you experince this and you're confident about nothing else is connecting to the endpoint just these ~200 systems there must be an error in the implementation of the clients themselves. My hunch is they're opening new connections instead of re-using the same channel for communications.

Can you check that somehow?

Zoltan Varadi
  • 41
  • 2