1

Due to some bug, my app is consuming too much memory and heap space is running out.

However, instead of requests failing, I'm getting an infinite hang and only following error in console:

*** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message can't create name string at JPLISAgent.c line: 826

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "http-nio-8090-ClientPoller"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Catalina-utility-2"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Catalina-utility-1"

Why are requests hanging and not failing? Is the only solution (besides not reaching out of memory to begin with) is to kill the service? (Its running in a container so it will get restarted)

My code to kill the service:

static class GlobalThreadExceptionHandler implements UncaughtExceptionHandler {
    @Override
    public void uncaughtException(Thread thread, Throwable throwable) {
        logger.error(String.format("Caught unhandled exception in thread %s", thread), throwable);
        Runtime.getRuntime().halt(137);
    }
}

public static void main(String[] args) {
  Thread.setDefaultUncaughtExceptionHandler(new GlobalThreadExceptionHandler());
  ...
}

(System.exit() was hanging as well)

I'm using Java 11 with Spring Boot version 2.1.6

Mugen
  • 8,301
  • 10
  • 62
  • 140

2 Answers2

1

This is the pause time or latency time for the application when Java garbage collector is trying to free memory. But seems garbage collector also is not able to run as memory is full. So application is hanged.

There are numbers of new garbage collectors like zgc, shenandoah that can run in parallel of the applications to continuously free memory but I don't know here they are applicable or not.

If you are using docker containers, you can check 'docker stats' to check when your contains uses and releases memory. And if your system has enough memory, you can increase/set MAX_HEAP size variable of your container accordingly. Also you can move/scale others contains accordingly so that this container gets more resources.

Suvendu Ghosh
  • 391
  • 1
  • 3
  • 16
0

I had a similar problem. I was already using zgc and / or shenandoah and it didn't help much.

My problem was that I was actually loading a lot of data into memory and the container ended up running out.

I was able to find the problem and optimize the code using Jprofiler (https://www.ej-technologies.com/products/jprofiler/overview.html). With it I was able to detect exactly which class was consuming memory.

Petter
  • 318
  • 2
  • 13