2

We're running an application on Linux using Java 1.6 (OpenJDK as well as Oracle JDK). The JVM itself has a maximum of 3.5 GB heap and 512 MB permgen space. However, after running a while top reports the process is using about 8 GB of virtual memory and smem -s swap p reports about 3.5 GB being swapped.

After running a bigger import of thousands of image files on one server, almost no swap space is left and calls to native applications (in our case Im4java calls to Image Magick) fail due to the OS failing to allocate memory for those applications.

In another case the swap space filled over the course of several weeks resulting in the OS killing the JVM due to being out of swap space.

I understand that the JVM will need more than 4 GB of memory for heap (max 3.5 GB), permgen (max 512 MB), code cache, loaded libraries, JNI frames etc.

The problem I'm having is how to find out what is actually using how much of the memory. If the JVM was out of heap memory, I'd get a dump which I could analyze, but in our case it's the OS memory that is eaten up and thus the JVM doesn't generate a dump.

I know there's jrcmd for JRockit, but unfortunately we can't just switch the JVM.
There also seem to be a couple of libraries that allow to track native memory usage but most of those seem to need native code to be recompiled - and besides Im4java (which AFAIK just runs a native process, we don't use DLL/SO-integration here) and the JVM there's no other native code involved that we know of.

Besides that, we can't use a library/tool that might have a huge impact on performance or stability in order to track memory usage on a production system over a long period (several weeks).

So the question is:

How can we get information on what the JVM is actually needing all that memory for, ideally with some detailed information?

Thomas
  • 87,414
  • 12
  • 119
  • 157
  • Check again. Looks like some native method invoked. And memory acquired from this native call. – talex Oct 13 '14 at 15:27
  • @talex Well, I know that the only native code our application is using is Im4java which states that it uses `ProcessBuilder`. We're also running on a JBoss 7.2 which might use native code as well but guessing if any part of that or which one is the culprit without further information is quite a pointless task IMHO. Hence the question for tools that provide that information, so that we're able to track down which native method causes that memory leak. – Thomas Oct 14 '14 at 07:31

1 Answers1

1

You may find references to "zlib/gzip" (pdf handling or http encoding since Java 7), "java2d" or "jai" when replacing memory allocator (jemalloc or tcmalloc) in JVM.

But to really diagnose native memory leak, JIT code symbol mapping and Linux recent profiling tools are required: perf, perf-map-agent and bcc.

Please refer to details in related answer https://stackoverflow.com/a/52767721/737790

Many thanks to Brendan Gregg

Yves Martin
  • 10,217
  • 2
  • 38
  • 77