What is the meaning of "AH00485: scoreboard is full, not at MaxRequestWorkers"?

Question

My Environment

CentOS 6.4 X86_64
Apache 2.4.4
PHP 5.4.16 (FPM)
2 Intel Xeon E5-2620 @ 2.00GHz (8 core, 16 threads in each processor)
48GB RAM registered memory.
3 Hard Disk 15RPM 145GB in RAID0 (by BIO

Interesting Variables

    <IfModule mpm_event_module>
        StartServers             2
        ThreadLimit             196
        MinSpareThreads         96
        MaxSpareThreads        192
        ThreadsPerChild         96
        MaxRequestWorkers      192
        MaxConnectionsPerChild   96
    </IfModule>

Apache Server Status

Server Version: Apache/2.2.4 (Unix) OpenSSL/1.0.1e mod_fastcgi/mod-fastcgi-SNAP-0910052141
Server Built: May 24 2013 16:48:07

Current Time: Monday, 17-Jun-2013 09:48:11 COT
Restart Time: Monday, 17-Jun-2013 08:35:14 COT
Parent Server Config. Generation: 1
Parent Server MPM Generation: 0
Server uptime: 1 hour 12 minutes 57 seconds
Server load: 0.05 0.10 0.09
Total accesses: 14144 - Total Traffic: 349.7 MB
CPU Usage: u.28 s.25 cu0 cs0 - .0121% CPU load
3.23 requests/sec - 81.8 kB/second - 25.3 kB/request
1 requests currently being processed, 191 idle workers
  PID | Connections       | Threads     | Async connections
      | total | accepting | busy | idle | keep-alive | closing
  ==============================================================
18997 | 3     | yes       | 1    | 95   | 0          | 3
18485 | 0     | yes       | 0    | 96   | 0          | 0
  ==============================================================
Sum   | 3     |           | 1    | 191  | 0          | 3

Error Log

The error message is

[Mon Jun 17 09:32:45.680842 2013] [mpm_event:error] [pid 8574:tid 140185091581760] AH00485: scoreboard is full, not at MaxRequestWorkers

This appears every few seconds. I don’t understand it. How can I fix it?

score 21 · Accepted Answer · edited Mar 14 '16 at 19:23

We had the same problem on Apache 2.4.6. After monitoring the server and adjusting the setting for several hours it appears to us that Apache may have a bug. What appears to happen is that the server processes occasionally goes into the G state (Gracefully finishing) and restarts to accept new requests, that's normal. What is not normal is that for some reason this can take up to a few minutes to restart. If you only have a few server process running and they all go into the G state at the same time then your scoreboard fills up and you won't be able to server any more requests.

What we did was increase the number of servers so there is a less of a chance that they will all go into the G state at the same time. Also make sure you allocate at least 25 threads (MaxRequestWorkers) for each server process because that appears to be the default (i.e. if 5 Servers x 25 ThreadsPerChild = 125 MaxRequestWorkers). You can change ThreadsPerChild if you like, we left it at default. If you don't allocate enough threads the additional servers will not start. We left MinSpareThreads at the default value which is 25 and the default for MaxSpareThreads which is 75. If you do modify these settings, the value for MaxSpareThreads must be greater than or equal to the sum of MinSpareThreads and ThreadsPerChild. Also MaxRequestWorkers must be equal to or less than the ServerLimit.

Here is what worked for us but it might not be the best configuration for you.

StartServers 3
MinSpareServers 5
MaxSpareServers 10
ServerLimit 250
MaxRequestWorkers 250
MaxConnectionsPerChild 1000
KeepAlive Off

Edit: This is a confirmed bug in httpd's mpm_event module which might not be fixable through configuration.
The linked bugtracker entry has a presumed patch and more discussion about how to fix this until a new version of the event module is officially released.

Your `MaxConnectionsPerChild` setting is way too low for production use. Furthermore, setting it to anything other than 0 is only meant to be done on Windows because it leaks memory internally. — rustyx, Oct 28 '15 at 13:55
Apache error_log also gives hints: `MaxRequestWorkers of 40 is not an integer multiple of ThreadsPerChild of 25, decreasing to nearest multiple 25` — dhaupin, May 20 '16 at 20:32
MaxSpareServers/MinSpareServers aren't applicable to mpm_event. I'm not sure what you meant here because the numbers are way too low to be MaxSpareThreads/MinSpareThreads. — Hamish Moffatt, Nov 10 '16 at 05:35
Also faced this trouble on Debian at Apache2 log rotation. Refer to https://support.plesk.com/hc/en-us/articles/213901685-Apache-crashes-scoreboard-is-full-not-at-MaxRequestWorkers — Yves Martin, Jul 17 '17 at 07:45
The patch mentionned in this answer was merged in 2.4.25. I'm here because I have the problem, although I'm using 2.4.25. Apparently, it appeared on a reload triggered by logrotate and the processes continue to write in `error.log.1`. `error.log` only mentions the reload. — Jérôme, May 15 '18 at 08:38

score 3 · Answer 2 · answered Mar 17 '15 at 11:55

Seeing the same issue.

Apache 2.4.7-1ubuntu4.4 on Ubuntu 14.04
Server Version: Apache/2.4.7 (Ubuntu)
Server MPM: event
Server Built: Mar 10 2015 13:05:59

We particularly can cause this behaviour by reloading apache.

What we then see, are a couple of old processes that don't stop:

root     28192  0.0  0.8 103772  8648 ?        Ss   Mar16   0:03 /usr/sbin/apache2 -k start
www-data  2530  0.3  2.1 865188 21516 ?        Sl   06:26   0:54  \_ /usr/sbin/apache2 -k start
www-data  2531  0.2  2.1 865436 21892 ?        Sl   06:26   0:51  \_ /usr/sbin/apache2 -k start
www-data  3299  0.3  2.0 864140 20628 ?        Sl   06:46   0:51  \_ /usr/sbin/apache2 -k start
www-data  7305  0.3  2.1 865100 21504 ?        Sl   08:36   0:37  \_ /usr/sbin/apache2 -k start
www-data 11952  0.2  1.8 863004 19268 ?        Sl   10:46   0:06  \_ /usr/sbin/apache2 -k start
www-data 13284  0.0  0.6 103772  6692 ?        S    11:18   0:00  \_ /usr/sbin/apache2 -k start
www-data 13553  2.1  2.0 866156 21248 ?        Sl   11:23   0:01  \_ /usr/sbin/apache2 -k start

Notice the 'older' and 'newer' PID's and start times. ^^

PID Connections     Threads Async connections
total   accepting   busy    idle    writing keep-alive  closing
7305    14  no  0   0   0   0   0
2530    13  no  0   0   0   0   0
3299    7   no  0   0   0   0   0
13553   65  no  17  8   0   25  25
2531    15  no  0   0   0   0   0
11952   10  no  0   0   0   0   0
Sum 124     17  8   0   25  25

GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
GGGGGGGGGGGW_WWWW__W_W_W_WWWWWWW__WWGGGGGGGGGGGGGGGGGGGGGGGGGGGG
GGGGGGGGGGGGGGGGGGGGGG

not an answer but comment – Hayden Thring Nov 17 '21 at 01:38 — Hayden Thring, Nov 17 '21 at 01:38

score 0 · Answer 3 · answered Jan 12 '22 at 15:23

0

Apache/2.4.52 [Wed Jan 12 05:25:49.646258 2022] [mpm_event:error] [pid 1400:tid 47341877433408] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.

any idea?

answered Jan 12 '22 at 15:23

user949958

1

This does not really answer the question. If you have a different question, you can ask it by clicking [Ask Question](https://serverfault.com/questions/ask). To get notified when this question gets new answers, you can [follow this question](https://meta.stackexchange.com/q/345661). Once you have enough [reputation](https://serverfault.com/help/whats-reputation), you can also [add a bounty](https://serverfault.com/help/privileges/set-bounties) to draw more attention to this question. - [From Review](/review/late-answers/508732) – mailq Jan 12 '22 at 22:09

score -1 · Answer 4 · edited May 25 '21 at 11:06

mlissner's answer helped me since I had come here after first increasing LimitRequestFieldSize and then MaxRequestWorkers while trying to install OpenProject: https://openproject.com.

I decided to redo everything in the hopes that starting from scratch would fix the error. Lo and behold the setup proceeded to run the needed migrations which hadn't been done the first time around.

Therefore I too submit this to the canon in the hopes that it may help someone else as well.

score -1 · Answer 5 · answered Mar 01 '19 at 16:08

We started seeing this when one of our replica databases went offline and started timing out. This tied up a gazillion threads in Apache, apparently until things were rather broken and we started getting this message.

Probably not the normal case, but I submit this to the canon in hopes that it may help others that see this error.

What is the meaning of "AH00485: scoreboard is full, not at MaxRequestWorkers"?

My Environment

Interesting Variables

Apache Server Status

Error Log

5 Answers5

Linked