1

I have more than one java application running in separate docker containers. I am trying to collect monitoring data such as GC log, thread dump, heap dump from the java process running inside a container using tools like jstat, jstack, jmap. Is it possible to capture this information from the host(outside containers)?

I am new to the containerized world. I understand that PID namespace of the host and container is different. When I execute jstack <PID> > thread_dump.txt from the host, it shows error message: Unable to open socket file /proc/root/tmp/.java_pid: target process doesn't respond within 10500ms or HotSpot VM not loaded

Where PID is process id from the host PID namespace.

When I execute jstack inside container ( docker exec -it <container_id_or_name>) then it is able to capture thread dump.

Where PID is process id from the container PID namespace.

Any hints on how to solve this?

Mahesh
  • 103
  • 1
  • 10

2 Answers2

4

jattach tool serves exactly for this purpose. The project is free and open source.

Examples:

  • jattach <pid> threaddump (works like jstack)
  • jattach <pid> inspectheap (works like jmap -histo)
  • jattach <pid> jcmd GC.class_stats

where <pid> is Java process ID in the host namespace.

As a bonus, jattach works even if container runs JRE with no JDK tools (jstack, jcmd, etc.) installed.

apangin
  • 92,924
  • 10
  • 193
  • 247
  • Hi @apangin, Super! I tried to install ```jattach``` package on centos7 but it seems the package is not available. Kindly advise. – Mahesh Aug 31 '21 at 16:32
  • @Mahesh Download the [provided binary](https://github.com/apangin/jattach/releases/tag/v2.0) or build it yourself (it's just a single `make` command). – apangin Aug 31 '21 at 16:39
  • Thank you :) Let me try. – Mahesh Aug 31 '21 at 16:42
  • I executed the ```make``` command from the directory where the ```makefile``` is present. I got the following error: ```mkdir -p build cc -O3 -DJATTACH_VERSION=\"2.0\" -o build/jattach src/posix/*.c /bin/sh: cc: command not found make: *** [build/jattach] Error 127``` – Mahesh Aug 31 '21 at 16:58
  • I have installed ```gcc``` package as mentioned here https://askubuntu.com/questions/1095168/command-not-found-cc-make-error-127. Now when I execute ```make``` command it shows message "make: Nothing to be done for `all'." – Mahesh Aug 31 '21 at 17:04
  • Just download prebuilt binary on the [Releases](https://github.com/apangin/jattach/releases/tag/v2.0) page. – apangin Aug 31 '21 at 17:05
  • I am getting a "Failed to change credentials to match the target process: Operation not permitted" error when I executed the ```jattach threaddump``` command. Any idea? – Mahesh Sep 01 '21 at 08:07
  • @Mahesh To attach to Java processes running under different users, you need to start jattach under a privileged user (root). Try `sudo jattach threaddump` – apangin Sep 01 '21 at 08:46
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/236643/discussion-between-mahesh-and-apangin). – Mahesh Sep 01 '21 at 10:00
1

The command you should use is:

docker exec -i <container_id_or_name> <your_monitoring_command>

The command will be executed inside the container, but it will report its output (by -i) to the caller console.

Antonio Petricca
  • 8,891
  • 5
  • 36
  • 74
  • Hello @Antonio, Yes I am able to capture data if I execute ```jstack``` within the container. But I am looking for a solution to do it from an outside container. – Mahesh Aug 31 '21 at 13:25
  • The solution I provided you is out of the container. – Antonio Petricca Aug 31 '21 at 13:30
  • Here is my command ```docker exec -t jstack > thread_dump.txt```. I passed PID of a process from the host PID namespace and got "No such a process" message inside ```thread_dump.txt``` file. It is working fine if I use PID from the container's PID namespace. Is there any command to get the PID from the container? Currently, I am using ```ps``` inside a container and getting PID. – Mahesh Aug 31 '21 at 13:35
  • Try using `docker ps`. – Antonio Petricca Aug 31 '21 at 13:38
  • Hello Friend, ```docker ps``` is returning a list of running containers. – Mahesh Aug 31 '21 at 13:59