After ordering and setting up a new web server (64bit, 4 vCores, 6GB RAM, Ubuntu 14.04) with Apache 2.4.7 (and PHP7 fpm) as event MPM but during busier traffic hours Apache crashes. The error log is filled with hundreds of entries (even during less busier hours) like that:
[Thu Jul 21 17:51:16.633617 2016] [mpm_event:crit] [pid 9290:tid 139705492948732] (22)Invalid argument: ap_queue_pop failed
[Thu Jul 21 17:51:16.633622 2016] [mpm_event:crit] [pid 9290:tid 139705492948732] (22)Invalid argument: ap_queue_pop failed
[Thu Jul 21 17:51:16.633627 2016] [mpm_event:crit] [pid 9290:tid 139705492948732] (22)Invalid argument: ap_queue_pop failed
[Thu Jul 21 17:51:18.632095 2016] [mpm_event:alert] [pid 9313:tid 139705857640190] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Thu Jul 21 17:51:19.632842 2016] [mpm_event:alert] [pid 9337:tid 139705857640190] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
I already have by default high system limits via ulimit -a:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1546681
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1546681
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
The blog entry from Mozilla didn't help either as my limit is already way higher than they suggest. I also read to increase the privvmpages value but it's high too:
$ cat /proc/user_beancounters
Version: 2.5
uid resource held maxheld barrier limit failcnt
130922: kmemsize 173862701 175349760 9223372036854775807 9223372036854775807 0
lockedpages 0 0 1572864 1572864 0
privvmpages 3327626 3330707 3670016 3670016 31705
shmpages 92638 92638 9223372036854775807 9223372036854775807 0
dummy 0 0 9223372036854775807 9223372036854775807 0
numproc 279 363 850 850 0
physpages 356751 363453 0 1572864 0
vmguarpages 0 0 1835008 1835008 0
oomguarpages 121370 121370 1572864 1572864 0
numtcpsock 69 72 9223372036854775807 9223372036854775807 0
numflock 11 12 9223372036854775807 9223372036854775807 0
numpty 2 2 9223372036854775807 9223372036854775807 0
numsiginfo 0 63 9223372036854775807 9223372036854775807 0
tcpsndbuf 1368640 1445072 9223372036854775807 9223372036854775807 0
tcprcvbuf 1938112 1987264 9223372036854775807 9223372036854775807 0
othersockbuf 233512 241968 9223372036854775807 9223372036854775807 0
dgramrcvbuf 0 0 9223372036854775807 9223372036854775807 0
numothersock 148 150 9223372036854775807 9223372036854775807 0
dcachesize 61545508 61876256 9223372036854775807 9223372036854775807 0
numfile 2133 2280 9223372036854775807 9223372036854775807 0
dummy 0 0 9223372036854775807 9223372036854775807 0
dummy 0 0 9223372036854775807 9223372036854775807 0
dummy 0 0 9223372036854775807 9223372036854775807 0
numiptent 338 338 9223372036854775807 9223372036854775807 0
$ free -m
total used free shared buffers cached
Mem: 6144 1823 4320 361 0 1553
-/+ buffers/cache: 270 5873
Swap: 1024 0 1024
The site gets totally up to 500.000 requests over the day but it was running on an way older (Software/Hardware) and smaller server (half the resources) before without any (resource) issues. My mpm-event settings:
<IfModule mpm_event_module>
StartServers 5
ServerLimit 32
MinSpareThreads 60
MaxSpareThreads 200
ThreadLimit 64
ThreadsPerChild 25
MaxRequestWorkers 800
MaxConnectionsPerChild 20000
</IfModule>
The load is usually around 0.05 and the RAM isn't even used more than 10% often. Any idea what I am doing wrong?
(Small note: It's a service people use for free and I built in my free time.)
