8

I have a site that consists mostly of static html pages with occasional ajax requests. The site is running on Apache, ajax is handled by Tomcat.

If Tomcat becomes slow to respond (java cannot connect to a database server, or just taking a long time to process a request for any reason) - it brings the whole site down: all static html pages are taking very long time to load (same with images, css, js).

Now if I just manually stop the Tomcat everything is still working fine - the site is fast and responsive, just ajax requests are not working.

How can I make slow responding Tomcat to not use all Apache resources, so static pages would always work no matter what is happening with Tomcat? Responsive html pages are much more important than not working ajax in my case.

httpd.conf:

Timeout 120
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 15

<IfModule prefork.c>
StartServers       16 
MinSpareServers   10 
MaxSpareServers   40
ServerLimit      512 
MaxClients       512
MaxRequestsPerChild  4000
</IfModule>

workers.properties

worker.worker1.port=8888
worker.worker1.reply_timeout=120000
worker.worker1.socket_timeout=150000

server.xml

 <Connector port="8080" protocol="HTTP/1.1" 
               connectionTimeout="20000" 
               redirectPort="8081" />

<Connector port="8888" scheme="http" protocol="AJP/1.3" redirectPort="8889" minSpareThreads="100"  maxThreads="400" connectionTimeout="20000" acceptorThreadCount="2"/>
serg
  • 245
  • 1
  • 3
  • 7
  • I assume you are running your Apache and Tomecat instances on the same server, is this true? – Jesse Webb Jul 20 '12 at 17:37
  • @Jesse Yes, both on the same server and I would prefer it to stay that way. – serg Jul 20 '12 at 17:39
  • What's your Apache config: What worker model, how many threads? Timeouts for mod_jk set? BTW: This question seams to better fit on ServerFault.com – Robert Jul 20 '12 at 17:42
  • ...and of course the tomcat configuration... and give also some details on the situation where the problem arises: Is it under heavy load? Or does it also happen if just every request going to tomcat takes a long time. does it also take a long time to request, lets say some css or static image file alone? –  Jul 20 '12 at 17:47
  • @Robert Please see the attached config. – serg Jul 20 '12 at 18:05
  • @Arne It happens in both cases - sometimes high server load causes tomcat to become slow to respond, sometimes if it is a db connection the load drops to 0 and it is still slow. So whenever tomcat is slow to respond for whatever reason all other http connections to this server are very slow too - html, css, imges, everything. – serg Jul 20 '12 at 18:05

2 Answers2

9

if for whatever reason tomcat does not handle your ajax requests fast, this reduces the number of requests that your apache can handle. Tomcat is configured to handle 400 requests in parallel, and there is also a default acceptCount of 100. So your tomcat is able to eat up 500 requests - at least: jvm and platform dependant there may even more connection request queued.

worker.worker1.reply_timeout=120000
worker.worker1.socket_timeout=150000

..tells mod_jk to wait about 1.7 days (socket_timeout is in seconds) for socket operations and 2 minutes for single networks packets from tomcat. You should adjust these values, to let mod_jk return an error as early as possible if tomcat is slow.

Let's assume your ajax requests are typically processed within a second with outliers up to two seconds. After beeing processed, the response is sent back at once. Then one may set worker.worker1.reply_timeout=2500, just half a second more. socket_timeout may even be omitted, as it is just a rough value. socket_connect_timeout, that defines how long it may take to connect from apache/mod_jk to tomcat should be added to worker.properties and set to a very low value, e.g. 100. as both sit on the same server. See The Apache Tomcat Connector -Reference for more details.

Every request, that goes from apache to tomcat counts for what you configured with MaxClientsin httpd.conf. The more requests are stuck in tomcat the less may be processed by apache for static content. If you shutdown tomcat in that situation, static content is delivered fast again, as it frees up resources for request processing in mod_jk and apache.

You have configured prefork.cand worker.c in httpd.conf at the same time. I guess prefork.c is the active, as MaxClients is set to 512 and this would match your observations and my interpretation.. ;-)

Telling mod_jk to give up earlier on long running requests to tomcat might help a lot, but you should also think about adjusting the number of client requests handled by apache (MaxClients) and the number of requests that tomcat processes (<connector maxThreads=...) in parallel. These numbers have to be balanced to what happens during normal operations. Some tracing of page loads may be helpful to see in what proportion these values should be. The absolute value depends on your servers specs, network situation, number of clients etc.

If the absolute number of possible parallel requests is to low, users will complain about slow page loads, while you won't see your server used to capacity. If it's far to high, it will use more memory than really needed, even slow down, and will not recover fast from problems with sub systems - e.g. the database. If apache sends out far more requests to tomcat as it would process in time, you would see some of them timing out while others are processed in acceptable time. Starting out with similar values at apache and tomcat is no bad idea, as long as the timeout settings ensure that a slow or unresponsive tomcat is not a millstone on apache's neck.

Arne
  • 206
  • 1
  • 2
  • Great answer, but if you could please also recommend better parameter values or explain how to tune parameters (without guessing) that would be perfect. For example what should I do - increase `MaxClients` to 1000 or drop tomcat threads to 100? What symptoms would hint me that some parameter value is too low or too high? (for example what would happen if apache clients is set to 1 mil?) – serg Jul 20 '12 at 20:36
  • And you were right about prefork - it is the one that's active. – serg Jul 20 '12 at 20:51
  • I've added two paragraphs about finding values. – Arne Jul 20 '12 at 21:45
  • "Note that socket_timeout is in seconds, and socket_connect_timeout in milliseconds", you have a socket_timeout of almost 2days... – mikejonesey Oct 27 '16 at 15:49
0

You seem to have only defined one worker. That means Apache can only talk to Tomcat via that one instance. A normal JK configuration contains 8-16 workers. Have a look at the default JK configuration file, which you can download.

user207421
  • 1,010
  • 6
  • 16
  • according to [workers.properties configuration](http://tomcat.apache.org/connectors-doc/reference/workers.html) `connection_pool_size` should not be used with values higher than 1 for apache 2.x and 1.3.x. That is the default value and the real value will be detected during runtime. – Arne Jul 21 '12 at 08:04
  • @Arne I don't understand the relevance. He hasn't defined connection_pool size at all. – user207421 Jul 22 '12 at 01:47
  • Thought you were confusing it, as the number of workers (tomcat instances) is obviously one here. – Arne Jul 22 '12 at 10:16
  • @Aren, Yes, that's what I said. I don't know what you're talking about frankly. – user207421 Jul 23 '12 at 03:16
  • The number of workers must be one here, as there is obviously only one tomcat, so I thought you confused that with the connections_pool_size. But that should be one as well. See the link to the configuration, the very first sentence and definition of the connection_pool_size parameter. – Arne Jul 23 '12 at 05:58