0

I have problem with OOM Exception in high load multi-thread Java project.
I appreciate any help you can give.

Delails: Project is build on Java+Mysql as storage.

There is no evidence about additional RAM use in moment of application crash(no jumps in any monitoring tool).There is a lot of free memory
CPU and I/O is ok too.
Jstack tool not found any thread deadlocks.
No slow or error logs in Mysql.
In hs_err you can see thread_blocked state but I don't found any reason for this.
Java run args, system limits, free memory you can see below.

Here is error in hs_err_filepart(full file attached):

There is insufficient memory for the Java Runtime Environment to continue. Native memory allocation (mmap) failed to map 1633681408 bytes for committing reserved memory.
Possible reasons:
The system is out of physical RAM or swap space In 32 bit mode, the process size limit was hit
Possible solutions:
Reduce memory load on the system
Increase physical memory or swap space
Check if swap backing store is full
Use 64 bit Java on a 64 bit OS
Decrease Java heap size (-Xmx/-Xms)
Decrease number of Java threads
Decrease Java thread stack sizes (-Xss)
Set larger code cache with -XX:ReservedCodeCacheSize= This output file may be truncated or incomplete.

Out of Memory Error (os_linux.cpp:2627), pid=11980, tid=0x00007f27148c8700

JRE version: Java(TM) SE Runtime Environment (8.0_92-b14) (build 1.8.0_92-b14)
Java VM: Java HotSpot(TM) 64-Bit Server VM (25.92-b14 mixed mode linux-amd64 compressed oops)
Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

--------------- T H R E A D ---------------

Current thread (0x00007f27883c5800): VMThread [stack: 0x00007f27147c8000,0x00007f27148c9000] [id=12005]

Stack: [0x00007f27147c8000,0x00007f27148c9000], sp=0x00007f27148c7160, free space=1020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xabd65a] VMError::report_and_die()+0x2ba
V [libjvm.so+0x4fb4db] report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x8b
V [libjvm.so+0x91d713] os::Linux::commit_memory_impl(char*, unsigned long, bool)+0x103
V [libjvm.so+0x91dc69] os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0x29
V [libjvm.so+0x917f6a] os::commit_memory(char*, unsigned long, unsigned long, bool)+0x2a
V [libjvm.so+0x98c343] PSVirtualSpace::expand_by(unsigned long)+0x53
V [libjvm.so+0x98d748] PSYoungGen::resize_generation(unsigned long, unsigned long)+0xf8
V [libjvm.so+0x98c8a2] PSYoungGen::resize(unsigned long, unsigned long)+0x22
V [libjvm.so+0x989b7b] PSScavenge::invoke_no_policy()+0xf3b
V [libjvm.so+0x98a301] PSScavenge::invoke()+0x41
V [libjvm.so+0x941410] ParallelScavengeHeap::failed_mem_allocate(unsigned long)+0x70
V [libjvm.so+0xabf077] VM_ParallelGCFailedAllocation::doit()+0x97
V [libjvm.so+0xac6aa5] VM_Operation::evaluate()+0x55
V [libjvm.so+0xac4e7a] VMThread::evaluate_operation(VM_Operation*)+0xba
V [libjvm.so+0xac51fe] VMThread::loop()+0x1ce
V [libjvm.so+0xac5670] VMThread::run()+0x70
V [libjvm.so+0x91fad8] java_start(Thread*)+0x108

VM_Operation (0x00007f22963f34e0): ParallelGCFailedAllocation, mode: safepoint, requested by thread 0x00007f25e8105000

--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
0x00007f25a8052800 JavaThread "pool-27-thread-32" [_thread_blocked, id=26278, stack(0x00007f2147523000,0x00007f2147624000)]
0x00007f24dc07f000 JavaThread "pool-33-thread-55" [_thread_blocked, id=26277, stack(0x00007f2147624000,0x00007f2147725000)]
0x00007f25bc06b800 JavaThread "pool-28-thread-48" [_thread_blocked, id=26272, stack(0x00007f2147725000,0x00007f2147826000)]
0x00007f2568060000 JavaThread "pool-24-thread-39" [_thread_blocked, id=26267, stack(0x00007f2147826000,0x00007f2147927000)]
0x00007f25bc06a000 JavaThread "pool-28-thread-47" [_thread_blocked, id=26262, stack(0x00007f2147927000,0x00007f2147a28000)]

jvm_args: -Xmx16G -XX:ReservedCodeCacheSize=256M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/DDT/CPA -XX:-OmitStackTraceInFastThrow

Memory:
free -m
total used free shared buffers cached
Mem: 64298 37022 27275 5 593 9466
-/+ buffers/cache: 26962 37336
Swap: 15847 957 14889

System limits:

ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 256455
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 64000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 256455
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

MisterLotus
  • 1
  • 1
  • Is it reproducable? Have you considered upgrading from `1.8.0_92-b14` to a more recent version? – Kayaman Jan 26 '18 at 08:35
  • Is it reproducible? - No, at test server all OK, even with twice load as current production load. Have you considered upgrading from 1.8.0_92-b14 to a more recent version? - If it will help we of course can do this. But I must prove that upgrade will help. – MisterLotus Jan 26 '18 at 09:08
  • You shouldn't have to prove anything (or at least not that much), as updating from `92` to `162` is a micro release. It might be the least effort/most useful attempt, if you don't have any other leads. – Kayaman Jan 26 '18 at 09:11
  • This is at least an option, thank you. But i wish to understand why such crash happens. – MisterLotus Jan 26 '18 at 09:19
  • It's about the best advice you can hope for. You're running in deep waters here and while there are a lot of smart people here (a lot smarter than me), nobody can effectively remote-debug your system even though you did a fine job providing plenty of information. – Kayaman Jan 26 '18 at 09:29
  • Around 4K thread seems to be in blocked state. This seems to be the main cause of OOM. Did you find any I/O, service read timeout errors, etc in the application logs? What monitor tools you have in place? Usually, those tools can show if there is a lot of contention on some method, DB or service calls. Did you observed anything like that? Did the JVM crashed during startup or it was already running? I agree that upgrading the JDK to latest stable build is a good idea. But to prove that it solves the issue, you need to reproduce it. Is there any difference between test and Prod env? – Rahul Jan 26 '18 at 12:09
  • Did you find any I/O, service read timeout errors, etc in the application logs? - No What monitor tools you have in place? - Visual VM,Jet Profiler,Nmon. Did the JVM crashed during startup or it was already running? - Random crash during work, it can happens after 5 minute , or month. Is there any difference between test and Prod env? - Server hardware configuration, production more powerfull.OS, java version and application version the same. – MisterLotus Jan 26 '18 at 12:42

1 Answers1

0

I think you might be facing the below issue with 64 bit VM and compressed OOPS:

https://bugs.openjdk.java.net/browse/JDK-8187709

https://blogs.oracle.com/poonam/running-on-a-64bit-platform-and-still-running-out-of-memory

Community
  • 1
  • 1
Rahul
  • 637
  • 5
  • 16
  • Thank you, tried to fix using provided info. But it didn't work. But I got additional info: If I try to run another java program at server while primary Application is working that will cause OOM exception in another application. That could be evidence of OS problem rather then java. – MisterLotus Jan 30 '18 at 14:26