Tomcat's gc timestamp stops increasing when monitored by jstat -gc -t

Question

Running tomcat 7.0.32 on oracle hotspot jdk 1.7.0_7 I get this peculiar output.

I was hoping to pinpoint a memory leak, so I ran jstat -gc -t 2000. At first everything looks reasonable. I get a new line every 2-ish seconds. At some point though, the timestamp column starts getting weird. It returns after 1 second, then after 5 seconds, then 2, then 7, until it reaches timestamp 518058.9 when it finally says "Screwit. You got enough data son. Go work on it. Imma call it a day."

Werid? Why does this happen? I was forced to restart the instance.

There could have been database errors and c3p0 was crying DEADLOCK, however this shouldn't stop the jvm reporting gc utilization? Right? Actually - not stop reporting it, just stop increasing the timestamp.

jstat output:

Timestamp        S0C    S1C    S0U    S1U      EC       EU        OC         OU       PC     PU    YGC     YGCT    FGC    FGCT     GCT
   259712.9 11584.0 15424.0 9560.8  0.0   285056.0 160309.1  470016.0   163917.6  142912.0 123577.1   1494  123.712  17     31.547  155.259
   259715.1 11584.0 15424.0 9560.8  0.0   285056.0 160319.3  470016.0   163917.6  142912.0 123577.1   1494  123.712  17     31.547  155.259
   259717.1 11584.0 15424.0 9560.8  0.0   285056.0 160334.9  470016.0   163917.6  142912.0 123577.1   1494  123.712  17     31.547  155.259
....
   518016.3 21696.0 22080.0  0.0   15712.7 592064.0 433482.6  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518018.0 21696.0 22080.0  0.0   15712.7 592064.0 433482.6  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518023.7 21696.0 22080.0  0.0   15712.7 592064.0 433482.6  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518025.8 21696.0 22080.0  0.0   15712.7 592064.0 433482.6  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518026.7 21696.0 22080.0  0.0   15712.7 592064.0 433482.6  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518029.9 21696.0 22080.0  0.0   15712.7 592064.0 433482.6  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518031.1 21696.0 22080.0  0.0   15712.7 592064.0 433482.6  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518034.2 21696.0 22080.0  0.0   15712.7 592064.0 433482.6  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518036.4 21696.0 22080.0  0.0   15712.7 592064.0 433482.6  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518037.3 21696.0 22080.0  0.0   15712.7 592064.0 433482.6  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518040.8 21696.0 22080.0  0.0   15712.7 592064.0 433502.5  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518043.1 21696.0 22080.0  0.0   15712.7 592064.0 433536.3  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518045.7 21696.0 22080.0  0.0   15712.7 592064.0 433631.9  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518050.1 21696.0 22080.0  0.0   15712.7 592064.0 433685.9  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518054.4 21696.0 22080.0  0.0   15712.7 592064.0 433766.8  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518056.1 21696.0 22080.0  0.0   15712.7 592064.0 433844.9  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518057.6 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
   518058.9 21696.0 22080.0  0.0   15712.7 592064.0 433920.0  527296.0   495141.6  140224.0 126562.4   1811  185.571  18     50.010  235.581
....

What does a thread dump show? Is there load on the application when it fails? — Alastair McCormack, Mar 22 '14 at 08:02
Could never complete a thread dump. Even with force. =( The applications running on it are frozen solid though. — zoomix, Mar 22 '14 at 16:16
Silly question, but is your Max Heap < System RAM? What are your Java command line options? — Alastair McCormack, Mar 24 '14 at 13:54
I don't think so. The machine has 8 gigs and the heap is only 2 gigs. Commandline: -Xms256m -Xmx2g -XX:MaxPermSize=512m — zoomix, Mar 24 '14 at 19:48
IMHO, in a server environment -Xms and -Xmx should be the same. I doubt that's the source of your problem but it'll aid performance. I presume you have no other JVMs or significant processes running. Things to try: take regular thread dumps before problem arises, take Heap Dump before problem, check Tomcat thread pool (should be approx 50 threads per core), set c3p0 connections to <90% threads. — Alastair McCormack, Mar 24 '14 at 20:17
Actually I watched a presentation on GC, on infoq I think, where the dude was enthusiastically encouraging Xms < Xmx from jvm 1.7. The jvm can better compensate for real life memory usage. Interesting about tomcat threads and c3p0 though. It's the only thing I don't feel I have covered.. — zoomix, Mar 25 '14 at 07:47
What happens if you try to get the timestamps using `-XX:+PrintGCDetails -XX:+PrintGCTimeStamps` ? — ansjob, May 09 '14 at 23:32
I couldn't tell you. Ultimately I chickened out, updated the tomcat & java and employed a restart scheme. Our customers had very little understanding for keeping a crashing server going while I investigate this one peculiar thing.. — zoomix, May 11 '14 at 13:24

Tomcat's gc timestamp stops increasing when monitored by jstat -gc -t

0 Answers0