Tuning Tomcat java app with Apache reverse proxy and Postgresql

Question

Looks like my apache reverse proxy may be acting as the bottleneck and bogging down the number of requests users can make.

I run Apache and Tomcat on server1, which has 96 gb RAM, rhel5 x64, 24 cores. Pretty beefy.

The Postgresql database runs on server2, with 50gb ram, rhel5 x64, 24 cores.

Tomcat is configured to use 60gb ram (I can increase and decrease this as needed). I am handling hundreds of hits per minute (both from users as well as remote API calls) and looks like Apache cannot keep up.

What settings should I boost in the httpd.conf to allow it to make sure of additional memory? I've also read that number of threads in apache correlates directly with the database threads. My postgres server is definitely beefy and is not used to its potential, but I don't want to increase one at the expense of the other. So how do I figure out the correct settings for both?

Apache is handling my SSL, serving on 443. Tomcat has a connector with port 8080.

Should I be increasing the number of workers for apache? Or giving them more memory?

@NeilMcGuigan would it make sense not to use Apache and instead use Tomcat standalone? Or are you suggesting something else such as nginx? — Nick, Jun 03 '15 at 21:13
@FedericoSierra yes, I've tried using both AJP and HTTP1.1 and performance wise, it was the same for me. — Nick, Jun 03 '15 at 21:13

score 2 · Answer 1 · answered Jun 03 '15 at 19:47

You are only guessing. Did you increase max connections on Tomcat and Apache? Did you increase the number of connections in you Tomcat apps DB connection pool and the max connections in PostgreSQL? Try increasing these 4 values and see what happens.

Also Tomcat with 60GB ram isn't a good idea. Every once in a while every Java garbage collector (it doesn't matter which one) does a stop the world pause. The time of this pause is proportional to the amount of ram. Your system will be completely unresponsive during this time. With 60GB ram this pause could be as long as 60 seconds. Try spawning multiple smaller instances of Tomcat and use mod_jk load balancing.

Good luck

Thanks, that is a very interesting point. The Java application is JIRA. I don't think I can spawn smaller instances of Tomcat. How would it make sense to figure out what the right amount of memory is? I've set max and min to 60gb and perm space at 1gb. — Nick, Jun 03 '15 at 21:26

score 0 · Answer 2 · answered Jun 04 '15 at 01:11

Enable GG logging on the Tomcat server and monitor your memory usage. I aim for a new gen GC every 10 to 15 seconds. For an online application, you may want to consider using concurrent GC. I aim for and old generation GC every hour or two, more frequently if the application stopped time is too long.

I would be surprised if Apache can't handle the load. Monitor the number of active connections. If you are running out, then you may need to increase number of children and/or number of connections per child. HTTP 1.1 can increase the number of required connections. If you have increased the keep alive timeout, consider reducing it.

Tuning Tomcat java app with Apache reverse proxy and Postgresql

2 Answers2