0

I've noticed recently that a VPS that I administrate has some strange Apache behaviour. Essentially, any HTTP request hangs for about a second before it is actually served.

Output from ab shows something like this:

$ ab -n 10 -c 1 http://server-public-ip/path-to-static-file
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
<snipped>

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   3.5      0      11
Processing:     2 1373 1784.6   1000    4854
Waiting:        2 1372 1784.6   1000    4854
Total:          2 1374 1784.4   1011    4854

<snipped>

My initial thought was that my mod_perl website was running slowly, but this is running on a statically-served file! My virtual host config looks like:

<VirtualHost server-ip:80>
    <Location />
            DefaultType text/html
            Order deny,allow
            Allow from all
            SetHandler perl-script
            PerlHandler HTML::Mason::ApacheHandler
            PerlSetVar MasonDataDir "/var/tmp/mason-cache"
            PerlSetVar MasonCompRoot "<path-to-comproot>"
            ReWriteEngine on
            ReWriteRule ^(.*)/$ $1/index
            ErrorDocument 404 /not_found
    </Location>

    <Location /static>
            Order deny,allow
            Allow from all
            SetHandler default-handler
    </LocationMatch>
</VirtualHost>

There is plenty of memory free (at least 32MB entirely free, i.e. not used for cache), lots of processor resources (load average is 0.00 on 1/5/15-minute windows), lots of disk I/O available (median disk transfer is less than 5 KB/s) . . . I've no idea what the problem might be. After the transfer begins it runs at full speed (more than 2MB/s, which is about the limit of my local connection.)

I'm using Debian stable, running a 2.6.32 kernel and Apache 2.2.16 (using prefork). There is little to no traffic going to the server, and I've tried tweaking the prefork settings in both directions (lots of spare workers to no spare workers).

Any thoughts?

kestrel
  • 163
  • 1
  • 8

1 Answers1

1

Keep in mind that even if you have separate handlers for different paths, the prefork children get reused based on your MaxRequestsPerChild setting. This means that a child may end up servicing a request to the perl handler before it services one to the static content (especially if you've got a load balancer with a health check against the perl handler). If there's a quirk in the Perl that is gumming up the works, it can gum up the static content on subsequent requests.

For this and other reasons, at a prior workplace we had a practice of isolating the static content to its own httpd process and using the load balancer to split traffic between the two based on URI context. The second webserver never even loads mod_perl, isolating any problems introduced by code to the other process.

Andrew B
  • 32,588
  • 12
  • 93
  • 131
  • You're correct, even just loading the mod_perl module slows down everything to mirror this behaviour. Very odd . . . I'll check my mod_perl configuration! – kestrel Feb 27 '13 at 22:43
  • Hmm. Can't reproduce my earlier results when disabling mod_perl; disabling it now seems to have no effects, everything is still slow. Gah. – kestrel Feb 27 '13 at 23:51
  • I ended up replacing the mod_perl script with nginx and a FastCGI wrapper, which works at the expected speed. I still have no idea what the problem might have been. Thanks for the advice, though, I'll definitely keep this in mind next time I set up a mod_perl server. – kestrel Mar 10 '13 at 20:41
  • Sorry that this wasn't more helpful. Glad you found something that worked though! – Andrew B Mar 11 '13 at 03:56