4

I'm running a heavy db-based GWT application on a debian VPS using tomcat server 7 (& JRE 1.6). My app contains a lot of java servlets which communicate with MySQL5 database via a tomcat connection pool (without connection pooling mysql will crash within less than 3 minute!)
My app works nice while there is no heavy load on tomcat server, but when number of online users & their requests increase, tomcat server fails with no useful log or error message(I just get a connection timeout error when I want to access any of web applications running on tomcat) & this problem exists till I reset my tomcat server. I know that I've NO memory limitation on my VPS nor any MySQL connection problem, so I really don't know that causes this situation :(
This is host tag in server.xml (I've many of these virtual hosts, but the host tag of all is similar)

<Host name="sub1.mydomain.com" appBase="/var/www/sites/gwt_app" >

    <Context path="" reloadable="true" docBase="myDocBase" 
     xmlValidation="false" xmlNamespaceAware="false" crossContext="true" >

    <Resource name="jdbc/mysql/db" auth="Container" type="javax.sql.DataSource"

    initialSize="3" maxActive="50" maxIdle="10" 
    maxWait="15000" removeAbandoned="true" removeAbandonedTimeout="120"
    validationQuery="select now();"

    username="user_1" password="pass" driverClassName="com.mysql.jdbc.Driver"
    url="jdbc:mysql://localhost:3306/db_1?useUnicode=true&amp;characterEncoding=UTF-8&amp;characterSetResults=UTF-8&amp;connectionCollation=UTF8_PERSIAN_CI&amp;noAccessToProcedureBodies=true"
    />
    </Context>

    <Valve className="org.apache.catalina.valves.AccessLogValve" directory="/var/www/sites/gwt_app_logs"  prefix="tomcat_access_" suffix=".log" pattern="common" resolveHosts="false"/>
</Host>

any idea?

Ehsan Khodarahmi
  • 4,772
  • 10
  • 60
  • 87
  • 2
    When you say Tomcat "crashes", what actually happens? Do you get exceptions in the error log(s)? If so, what are they? Does the JVM fail and the process dies? If so, did you get a core dump or hs_* file? If Tomcat is still running but not responding to requests, please take a thread dump (or several) and post them somewhere. Also, posting your and configuration (minus any sensitive information such as passwords, etc.) will be helpful. – Christopher Schultz Jun 03 '12 at 16:05
  • I'll ask again: what happens when "Tomcat server fails"? If the process is still running, you need to take some thread dumps and post those so we can see what the server is actually doing. – Christopher Schultz Jun 07 '12 at 18:56
  • 1
    jstack/jmap are your friends, use them – bestsss Jun 08 '12 at 12:01

4 Answers4

4

You need to provide more details about your resources utilization. But based on your description you might be having an issue HTTP thread pool being exhausted. By default (with BIO connector) Tomcat can handle only 200 concurrent connections and AFAIK can keep that many connections "on hold" in the backlog queue.

This means that only 200 connections can be handled at the same time and another 100 will wait for the pool. 301st connection will be rejected or timeouted.

Here are pointers what to examine:

  • HTTP thread pool utilization
  • backlog queue size
  • garbage collection activity

Tomcat provides valuable metrics for that. You might also find my article about squeezing more concurrent connections from Tomcat useful.

Tomasz Nurkiewicz
  • 334,321
  • 69
  • 703
  • 674
  • 1
    This is not correct. Tomcat 7.0.x can handle more than 100 concurrent connections by default. The BIO connector with default settings has 200 maxThread PLUS 100 in the acceptCount/backlog queue. Using the NIO connector, it is very possible to scale to 1000s of concurrent connections. – Pidster Jun 04 '12 at 15:09
  • @Pidster: +1, that's right, I said *by default* (i.e. with BIO connector) - however I recalled incorrectly that default size of the thread pool is 100, I trust you that it is actually 200 (corrected). Thanks! – Tomasz Nurkiewicz Jun 04 '12 at 15:55
  • @TomaszNurkiewicz Can you please provide a link to support "**only 200 connections can be handled at the same time**"? – Dinesh Singh Sep 05 '13 at 18:44
1

A recent bug in the BIO connector may be a factor here.

See: https://issues.apache.org/bugzilla/show_bug.cgi?id=53186

However, if your application also crashes in under 3 minutes with DB connection pooling, there is likely to be something wrong with your application or DB handling code.

Pidster
  • 628
  • 4
  • 9
1

You may change CATALINA_OPTS parameter in /TOMCAT_DIR/bin/catalina.sh for Concurrent garbage collection 

CATALINA_OPTS="-XX:+UseConcMarkSweepGC -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -Xloggc:/TOMCAT_DIR/logs/gc.log"
hcg
  • 642
  • 5
  • 8
0

use tomcat clustering and apache network load blanc

yacine ouah
  • 345
  • 1
  • 4
  • 6