0

The machine(linux) runs three servers wating for TCP connections, writing in JAVA. After they runs steadily for about two weeks, the system load suddenly raises utterly high to 30.05, 12.93, 9.70(during the two-week normal running, it is often 0.0 , 0.0, 0.0), and it flaps sharply.

I log on to it, check by vmstat, found nothing specical, except that memory usage is a bit of abnormally high.

So I restart the three java servers and it goes well again.

Are there any tools or methods to figure out what exactly causes this problem if I encounter it again?

l4rmbr
  • 113
  • 1
  • 3
  • 1
    As a good developer have you traced and profiled your own code before suspecting the server? It can take weeks for excessive memory usage or errors in the code to surface. – Brent Pabst Oct 11 '12 at 01:58
  • And during those days, weeks or whatever you should be seeing the memory used by Java gradually increasing. You don't have to wait for the system to screw up to diagnose this. – John Gardeniers Oct 11 '12 at 02:57

1 Answers1

1

Java, high-memory usage after 2 weeks ... smacks of memory leak, and resulting swapping (I/O bound, which might explain the load quite easily). If, as you say, vmstat looks normal (you checked si / so?) it could be that some other form of comms between java process and either clients or each other aren't being handled quickly enough any more.

tink
  • 1,035
  • 11
  • 20
  • Yes, sounds like a application issue, potentially caused by underlying hardware constraints but still something fishy at the application layer. – Brent Pabst Oct 11 '12 at 02:22