3

I wrote a RESTful web service to be the backbone behind a mobile app using PHP and the Restler library. It's running on my dev server running Windows Server 2008 R2, PHP 5.3.5, Apache 2.2.17, and MySQL 5.5.8. Just for giggles I decided to benchmark my dev server and ran into a possible configuration issue

If I run ab -k -n 1000 -c 50 http://myproductionserver.com/something/restful through the windows CLI I usually get stopped between 300-700 requests. The error says "apr_socket_recv: An existing connection was forcibly closed by the remote host. (730054). Total of X requests completed."

I figured I would check my Apache Error logs. Not Helpful. Here's the error I invoked with ab:

[Fri Feb 24 17:23:27 2012] [notice] Parent: child process exited with status 255 -- Restarting.
[Fri Feb 24 17:23:27 2012] [notice] Apache/2.2.17 (Win32) PHP/5.3.5 configured -- resuming normal operations
[Fri Feb 24 17:23:27 2012] [notice] Server built: Oct 18 2010 01:58:12
[Fri Feb 24 17:23:27 2012] [notice] Parent: Created child process 768
[Fri Feb 24 17:23:28 2012] [notice] Disabled use of AcceptEx() WinSock2 API
[Fri Feb 24 17:23:28 2012] [notice] Child 768: Child process is running
[Fri Feb 24 17:23:28 2012] [notice] Child 768: Acquired the start mutex.
[Fri Feb 24 17:23:28 2012] [notice] Child 768: Starting 64 worker threads.
[Fri Feb 24 17:23:28 2012] [notice] Child 768: Listening on port 80.

So when I got home, I ran the same test using my local ip- thinking it might be a network issue. Same thing happened. I've been searching for quite a while with not a lot of answers. I found a very similar problem and the resolution was to use "Win32DisableAcceptEx" in the httpd.conf file saying the issue is actually a bug with PHP .dlls. I disabled all of them but the basic MySQLi. As you can tell from the logs above, neither did much of anything.

Anyone have any ideas? Am I seriously crashing the server with 1000 requests? Let me know if you need more info! I know it's just a dev server sitting at home- but it mimics my production server. I'll need to have this sorted out before moving forward. I appreciate the assistance!

Zoredache
  • 130,897
  • 41
  • 276
  • 420
Kevin
  • 1,403
  • 2
  • 11
  • 9
  • I think you also want to set at least "ab -v 3" to see what the response codes are from apache. 408 indicates a request timeout. But as you have also set -k for keepalives ab is expecting to reuse those 50 connections, but apache will force close the keepalive if it is idle for 5 seconds. So there might be some odd interaction causing that. try it without -k setting and see what happens, also try the -r flag to ab with -v set to 3 or 4, as without -r ab is exiting on the first error – Tom Feb 25 '12 at 02:45
  • All 200s then "Send request failed!" and the same error message. that was with -v 3 and -k flags set. Without -k I get 100 requests before failure, and all 200s leading up to it. With -r set, i got 880 / 1000 requests completed, and it would spam the same error message periodically. With -r set and -v not set I get "apr_poll: an operation was attempted on something that is not a socket".-v 4 and -r is the same error. – Kevin Feb 25 '12 at 03:10
  • I would start from known good setting for ab, and increase the concurrent connections and total until you start to see errors. – Tom Feb 25 '12 at 03:24
  • try ab -c5 -n100 http://" and work up, try any find what the level is for concurrent requests. There is a msdn article here (http://blogs.msdn.com/b/malarch/archive/2006/06/26/647993.aspx) which gives a scenario in which the SynProtection mechanism which is used to protect against "syn DOS" attacks, can fool the client into thinking its connected to the server, when its not actually, and results in the sort of weird connection errors you are seeing. – Tom Feb 25 '12 at 03:36
  • Hmm. The ab -c5 -n1000 command worked great the first time. When I bumped the level of -c up it failed immediately, and now -c5 is getting the "connection forcibly closed" error. – Kevin Feb 25 '12 at 05:40
  • C1 is the only setting that runs consistently- seems like it's a concurrent request issue. I'll make sure the MySQLi queries are locking the tables properly and update with any progress. If you have any more input, I would greatly appreciate it! – Kevin Feb 25 '12 at 05:51

1 Answers1

3

Its likely that you would have to look at the MaxClients setting and the ServerLimit settings for your apache instance, which might be lower than you might expect to be sure its not an apache limit being imposed. However a quick look at the docs suggest that the default for both is 256, which is less that your ab imposed 50 connections limit.

Is the server actually crashing, ie you need to restart httpd or does it continue to serve requests once the load has been removed?

I would set loglevel debug and httpd.conf, and try with fewer connections and total n, and ramp up the test. I would also look and see what the values for RLimitCPU
RLimitNPROC
RLimitMEM

are as they are also limits at which apache might start killing processes/threads/connections etc.

Tom
  • 11,176
  • 5
  • 41
  • 63
  • I added the following to my httpd.conf: ServerLimit 1024 StartServers 10 MinSpareServers 10 MaxSpareServers 20 MaxClients 1024 MaxRequestsPerChild 0 It seems to have helped. I get through the test once in a while with mostly failures due to "Length". To answer your question, it continues serving requests shortly after it's self invoked restart. Checking the values right now will update. – Kevin Feb 25 '12 at 02:33
  • Those three- RLimitCPU, RLimitNPROC, RLimitMEM aren't even in my httpd.conf so im assuming the default values are taking over. – Kevin Feb 25 '12 at 02:37
  • I tried testing with a few values, now I don't even get past 150 requests. Any recommended settings? – Kevin Feb 25 '12 at 02:50
  • I think start by running with the verbose option `-v 3` you want to see the error that apache is returning. – Tom Feb 25 '12 at 03:03
  • Tom - This was actually a "ThreadsPerChild" issue as the default is something relatively low. I bumped it up to 250- no problems. I appreciate your assistance! – Kevin Feb 27 '12 at 21:13
  • @kevin glad if I helped in any way. If nothing else these types of problems expose some of the internals of how apache deals with requests, which is quite interesting. Well if you are interested in that sort of thing... ;-) – Tom Feb 28 '12 at 12:54
  • 1
    Well, you go the ball rolling and i'm grateful for that. I'll up vote your answer when I get 15 rep here. All my rep is on stackoverflow.com. It's very interesting information- especially now that i'm writing web services it has become even more important to understand Apache's internals. – Kevin Feb 28 '12 at 21:49