jvmkill is unable to kill the JVM in an image created by Paketo Java Buildpack

Question

I have a spring boot application v2.4.3 in a container created using the official spring boot gradle plugin which uses buildpakcs to do so.

One of the layers is jvmkill:

[creator]     Adding layer 'paketo-buildpacks/bellsoft-liberica:jvmkill'

which is perfectly fine and it is adding the jvm arg for jvmkill properly

Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx387804K -XX:MaxMetaspaceSize=148771K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 1G, Thread Count: 250, Loaded Class Count: 23852, Headroom: 0%)                                                                                                                                           
Adding 129 container CA certificates to JVM truststore                                                                                                                               
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -agentpath:/layers/paketo-buildpacks_bellsoft-liberica/jvmkill/jvmkill-1.16.0-RELEASE.so=printHeapHistogram=1 -XX:ActiveProcessorCount=2 -XX:MaxDirectMemorySize=10M -Xmx387804K -XX:MaxMetaspaceSize=148771K -XX:ReservedCodeCacheSize=240M -Xss1M                                                                                                                         
   .   ____          _            __ _ _                                                                                                                                              
  /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \                                                                                                                                             
 ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \                                                                                                                                            
  \\/  ___)| |_)| | | | | || (_| |  ) ) ) )                                                                                                                                           
   '  |____| .__|_| |_|_| |_\__, | / / / /                                                                                                                                            
  =========|_|==============|___/=/_/_/_/                                                                                                                                             
  :: Spring Boot ::                (v2.4.3)

The app is running in Kubernetes (AWS EKS) but when I got an OOM

java.lang.OutOfMemoryError: Java heap space

jvmkill kicks in, prints the heap dump and sends the kill signal

Heap                                                                                                                                                                                 
  def new generation   total 116736K, used 90606K [0x00000000e8400000, 0x00000000f02a0000, 0x00000000f02a0000)                                                                        
   eden space 103808K,  87% used [0x00000000e8400000, 0x00000000edc7bae0, 0x00000000ee960000)                                                                                         
   from space 12928K,   0% used [0x00000000ef600000, 0x00000000ef600000, 0x00000000f02a0000)                                                                                          
   to   space 12928K,   0% used [0x00000000ee960000, 0x00000000ee960000, 0x00000000ef600000)                                                                                          
  tenured generation   total 259456K, used 249951K [0x00000000f02a0000, 0x0000000100000000, 0x0000000100000000)                                                                       
    the space 259456K,  96% used [0x00000000f02a0000, 0x00000000ff6b7f18, 0x00000000ff6b8000, 0x0000000100000000)                                                                     
  Metaspace       used 74292K, capacity 76000K, committed 76824K, reserved 208160K                                                                                                    
   class space    used 8689K, capacity 9350K, committed 9600K, reserved 140576K                                                                                                       
 jvmkill killing current process                          ```

but the jvm is never killed. Inspecting the container, I notice that the app runs with PID 1 which cannot be killed (or signaled) from inside the container.

cnb@myhost-6968d47f4b-2cnlj:/$ ps -fea
UID        PID  PPID  C STIME TTY          TIME CMD
cnb          1     0  6 19:49 ?        00:00:54 java org.springframework.boot.loader.JarLauncher
cnb        121     0  0 20:02 pts/0    00:00:00 bash
cnb        131   121  0 20:02 pts/0    00:00:00 ps -fea

since all this has been built by the java buildpack itself, I would expect that it is aware that PID 1 cannot be killed and launch the app differently to work properly.

Is there something that I am missing or need to configure for buildpack jvmkill to work out-of-the-box?

Workarounds:

1. If I were running the image directly with docker, I can use docker run --init to be able to signal the jvm process (my app runs with PID 7). Not valid for Kubernetes.

cnb          1  4.0  0.0   1120     4 ?        Ss   20:10   0:00 /sbin/docker-init -- java etc...
cnb          7  4.0  0.0  18372  1580 ?        S    20:10   0:00 java etc...

2. I can shareProcessNamespace: true in my k8s spec to have it running with PID other than 1, but would be hard to justify due to security compliance requirements.

3. I could just add the jvm flag -XX:+ExitOnOutOfMemoryError but then don't have the nice heap dump that jvmkill shows plus jvmkill thread creation failure coverage.

Answer 1

Given the workaround mentioned in https://github.com/airlift/jvmkill#using-inside-docker-containers, you need something else to be pid 1. Kubernetes does not have the equiv of --init but if you include tini in your image, you can use it manually via the container command. You could also use sh similarly.

Also keep in mind this will only protect you from hitting the Java-level heap size limit. If you hit the actual container memory limit then your process is immediately terminated with no chance to catch it and do anything.