5

I have a weird issue. Every time I stop Tomcat, SSH becomes very slow, almost totally unresponsive. It takes a minute or even more for SSH to accept any command. When I finally get Tomcat started again everything goes back to normal.

This is happening on a very busy server. Problem is that I need to stop the Tomcat often because of the application upgrades. It would normally take few seconds to upgrade, but here it takes almost 10 minutes and because of that we are experiencing unwanted downtimes.

One thing that I see is when I stop Tomcat top shows a lot of ksoftirqd/X processes at 100% CPU. Could this be the issue?

Kernel version is: 2.6.18-308.11.1.el5

Red Hat version is: Red Hat Enterprise Linux Server release 5.9 (Tikanga)

Any idea why this is happening?

nickgrim
  • 4,466
  • 1
  • 19
  • 28
Paxxil
  • 181
  • 1
  • 10
  • Not an idea per se, but what you could try to investigate further is to check why the `ksoftirqd` kernel-threads are that busy. I suggest to install `atop`, run as `atop -l` and filter for `ksoftirq` (press "P" and give "ksoftirq" as filter). Then observe `RDDSK` and `WRDSK`, as well the state of the process (`S`) - is it non-interuptable - `D` - for instance? Also you might as well have a look at your io-subsystem using `iostat` to check for possible IO-bottlenecks. – Alexander Janssen Mar 20 '13 at 13:48
  • 3
    This is a shot in the dark here, but maybe set the firewall to drop all requests to the Tomcat port before you stop Tomcat. – user1936123 Mar 20 '13 at 21:10
  • Did you know when this "slow ssh" issue starts? Is something new in server environment? – dsznajder Mar 24 '13 at 11:59
  • 1
    Solution from @user1936123 worked. I haven't tried any other solution because we can not afford downtimes and it is very hard to test anything when console is almost unresponsive. – Paxxil Mar 28 '13 at 08:37
  • Smells like your app has a leak. You can get what all the interrupts are for by doing `cat /proc/interrupts` – user May 10 '13 at 16:31

2 Answers2

1

I know that this isn't the "best practice", but, i would suggest you to reset tomcat remotely through ssh and set the outputs to null:

ssh your_server '/etc/init.d/tomcat restart > /dev/null 2>&1'

(You could replace the command above with the equivalent you use to restart tomcat).

This is a workaround, not a solution. Could you try this while connected on other ssh session and check if the problem still occurrs and affects all the sessions?

1

Slow ssh connection / ssh lag is a symptom of a high load. High load is often caused by io blocking, which is often caused by swapping.

to check your load, run uptime or top you will probably see load numbers over 10 when ssh is not responding. They will probably hover under 2 during normal use.

run free or top to see your memory usage, you will probably see a lot of swap.

Once you find out the root symptoms, you can then search for why is "tomcat swapping on shutdown" or "tomcat high load" which is probably because it's trying to write some things stored in cache or swap to disk. Is your tomcat jvm max heap size larger than the amount of memory you have?

Maybe something connecting to your webapp retries constantly when it goes down creating DoS scenario.

This could all be specific to your webapp so use general terms.

MattPark
  • 303
  • 5
  • 20