1

we have a very high load System with round about 8.5 Million User per day. Actually we have 8 Apache2 Loadbalancer (Ubuntu 10.04.3 LTS x64 via DNS-RoundRobin) which are sending the traffic via AJP1.3 to 20 Workers. On this workers our self-written Grails-Webapplication is running on a tomcat7 + java7 server (Ubuntu 11.10 x64). The Application is Logging to a local postgresql 9.1 database. All Servers are hostet on Hetzner Germany an have an Intel® Core™ i7-2600 with 16GB's RAM an 2x3TB Raid 1 HDD.

At Prime-Time (6(pm) till 12(pm)) we have sometimes serious performance bottlenecks but the hardware is not the Problem. CPU is max at 50%, load max. 2.5 and RAM only used 3-4GB max. But i have no Idea, where is the Problem.

Here is an excerpt of my server.xml:

<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector  port="8009"
            protocol="AJP/1.3"
            maxThreads="5000"
            minSpareThreads="25"
            maxSpareThreads="75"
            acceptCount="200"
            redirectPort="8443"
            connectionTimeout="60000"
            debug="4"
/>

At peak the Tomcats handle 400-450 Accesses per second. Have anyone some tips how to optimize the performance of Apache + mod_jk + tomcat7?

Bart De Vos
  • 17,911
  • 6
  • 63
  • 82
Arny80Hexa
  • 139
  • 1
  • 4
  • 14
  • i would check tomcat's startup config as well. catalina opts and stuff – Alfabravo Dec 14 '11 at 10:28
  • We have set the ulimit till 65536 in Tomcat's Startup, because in the past we had problems with it. Java_Opts we have also edit: JAVA_OPTS="${JAVA_OPTS} -XX:+UseConcMarkSweepGC" JAVA_OPTS="$JAVA_OPTS -Xms512m -Xmx2048m -XX:MaxPermSize=256m". Catalina_opts are default at this Time. – Arny80Hexa Dec 14 '11 at 13:29
  • 1.) I would reduce access to tomcats putting static resources to be served by apacheHTTP itself; 2.)Would check GC with -XX:+PrintGCTimeStamps -XX:+PrintGCDetails; 3.) Depending on this, improve generation management with JVM (http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html) – Alfabravo Dec 14 '11 at 13:52
  • ConcMarkSweepGC can cause long delays if it fails to finish the concurrent GC in time and has to do a full stop-the-world collection instead. You really need to log your garbage collection and analyse the logs carefully. – Mike Scott Dec 14 '11 at 14:47

1 Answers1

1

Garbage collection would be my guess -- it can cause bottlenecks on loaded Java servers. Have you studied your garbage collection logs to get an idea of the delays involved? What are your garbage collection settings

Mike Scott
  • 7,993
  • 31
  • 26