Tools available for debugging production Issues in Java applications

Question

Just wondering what are various tools & techniques out there to debug production issues on Java applications. Like,

What are the ways and tools to take thread dumps?
What are the ways and tools to take heap dumps?
What are the tools to analyse the above dumps?

(Assumption all are in Linux/Unix environment)

I edited your post, as I am fairly sure you were interested in memory _dumps_, not memory _dumbs_ ;-) — Péter Török, May 28 '10 at 18:06
What is the difference between a memory dump and a heap dump for you? — Pascal Thivent, May 28 '10 at 19:39
Thank you guys, for pointing out all the typo's.. I have corrected my questions. — Java Guy, May 31 '10 at 01:13
Consider adding the details of the JVM you are using to the question. For Linux this might be Sun Java, OpenJDK, IBM Java, JRockit... — Thorbjørn Ravn Andersen, May 31 '10 at 03:57

score 7 · Accepted Answer · edited Jul 30 '12 at 00:54

What are the ways and tools to take thread dumps?

For a thread dump, you can use JConsole, VisualVM or, more simply, send a QUIT signal to the target process

kill -QUIT <pid>

or

kill -3 <pid>

Since Java 5, there is also jstack which is platform independent and has a nice -m option to print both Java and native frames (mixed mode).

What are the ways and tools to take heap dumps?

With Sun VMs, jmap, Sun JConsole, Sun VisualVM, SAP JVMMon. For IBM VMs, check this page. Actually, the Eclipse MAT wiki has a nice Getting a Heap Dump section summarizing all the options.

What are the tools to analyse the above dumps?

For thread dumps I use TDA - Thread Dump Analyzer (for Sun JDKs) and IBM Thread and Monitor Dump Analyzer (for IBM JDKs). Samurai is also very nice (it works like a tail -f and picks up thread dumps from your std/stderr automatically, it can also read "-verbose:gc" logs) and has been tested against VMs from Apple, BEA, HP, Sun and IBM (can also read IBM's javacore).

For heap dumps, I use VisualVM (for Sun JDKs) or IBM Heap Dump Analyzer (only for IBM JDKs) or the über awesome Eclipse MAT depending on my needs. The later is able to work with HPROF binary heap dumps (produced by Sun, HP, SAP, etc... JVMs), IBM system dumps (after preprocessing them), and IBM portable heap dumps (PHD) from a variety of IBM platforms).

Thanks! I will go through these links to get more info... – Java Guy May 31 '10 at 01:14 — Java Guy, May 31 '10 at 01:14

score 1 · Answer 2 · answered May 28 '10 at 18:08

Assuming JDK 6, Take a look at the following article to obtain thread dumps of a running program:

http://java.sun.com/developer/technicalArticles/Programming/Stacktrace/

You can use JHat to do heap analysis:

http://java.sun.com/javase/6/docs/technotes/tools/share/jhat.html

If you want to do memory dumps take a look at jmap:

http://java.sun.com/javase/6/docs/technotes/tools/share/jmap.html

Alternatively, if you need to do some more in depth analysis look at a profiler such as Yourkit:

http://www.yourkit.com/

score 1 · Answer 3 · answered May 28 '10 at 18:11

1

There's also these two things that can interest you:

answered May 28 '10 at 18:11

ewernli

38,045
5
92
123

There is not Ctrl-break under Linux... it's kill -3 all the way... – Jonathan Holloway May 28 '10 at 18:13
He is anyway not interested in the thread dump :) That was just to complement your answer. But thanks for pointing me that out. – ewernli May 28 '10 at 18:15

score 1 · Answer 4 · edited May 28 '10 at 18:31

1

There is no standard toolset for JVMs. These are vendor dependant, and you should consult the documentation.

For Sun Java 6 the VisualVM program is very, very helpful to get a quick profile and stack trace of a running program.

edited May 28 '10 at 18:31

John Topley

113,588
46
195
237

answered May 28 '10 at 18:29

Thorbjørn Ravn Andersen

73,784
33
194
347

score 1 · Answer 5 · answered May 30 '10 at 02:24

The tool I use for this kind of debugging a Sun JVM are

jstack to take a thread dump
jmap to take a memory/heap dump, or histogram
eclipse mat for post analysis of the heap dump produced by jmap
visual vm has a nice ui for live analysis of the vm (can also take heap and thread dumps)

score 0 · Answer 6 · answered May 28 '10 at 18:24

0

And I think the best way to debug java application in production environment is NOT dump and so on, but a good managment of logs.

See for instance slf4j.

answered May 28 '10 at 18:24

Istao

7,425
6
32
39

Huh? Thread dumps and Heap dumps are incredibly valuable and and while logs are very useful too, they won't tell you what threads (including the one of the server) were doing or what was in memory when you got this OOM. You can't do serious problems analysis with logs only (especially if you mean application logs). – Pascal Thivent May 28 '10 at 19:37
logs are primarily used for getting answers to questions you knew you would be asking later. Now, what about those questions you did _not_ think of then? – Thorbjørn Ravn Andersen May 31 '10 at 03:56

score 0 · Answer 7 · answered Aug 29 '10 at 19:15

To debug issues with memory allocations, InMemProfiler can be used at the command line. Live vs collected allocations can be tracked and collected objects can be split into buckets based on their lifetimes.

In trace mode this tool can be used to identify the source of memory allocations.

Tools available for debugging production Issues in Java applications

7 Answers7