We are running tomcat 9 with JVM 8 version. We have set min-heap as 2GB, Max is 7GB. But we are very often seen Native memory allocation (mmap) failed error. Heap Conf Looks like:
"-Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xms2048M -Xmx7168M -XX:MaxPermSize=512M"
I can see before failure there is enough heap then why it failing. Our System vm.max_map_count
sysctl vm.max_map_count
vm.max_map_count = 65530
free -m total used free shared buff/cache available Mem: 52187 19653 452 40 32082 32012 Swap: 4095 80 4015 .
I observed OS vm.max_map also keep on increasing , usually on linux default size is around 65k. yesterday it was 8k(when tomcat restarted) now it increased to 25k
pmap 46012|wc -l
33410
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux.cpp:2627), pid=51706, tid=0x00007f403bbbf700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_102-b14) (build 1.8.0_102-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.102-b14 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
--------------- T H R E A D ---------------
Current thread (0x00007f48d9131800): JavaThread "CBONLINE-DBAdapter-DB-Pool connection closer" daemon [_thread_new, id=68713, stack(0x00007f403babf000,0x00007f403bbc0000)]
Stack: [0x00007f403babf000,0x00007f403bbc0000], sp=0x00007f403bbbe9a0, free space=1022k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xac52aa] VMError::report_and_die()+0x2ba
V [libjvm.so+0x4fc29b] report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x8b
V [libjvm.so+0x922603] os::Linux::commit_memory_impl(char*, unsigned long, bool)+0x103
V [libjvm.so+0x9226cc] os::pd_commit_memory(char*, unsigned long, bool)+0xc
V [libjvm.so+0x91c17a] os::commit_memory(char*, unsigned long, bool)+0x2a
V [libjvm.so+0x92089f] os::pd_create_stack_guard_pages(char*, unsigned long)+0x7f
V [libjvm.so+0xa6a55e] JavaThread::create_stack_guard_pages()+0x5e
V [libjvm.so+0xa73d44] JavaThread::run()+0x34
V [libjvm.so+0x9249c8] java_start(Thread*)+0x108
C [libpthread.so.0+0x7ea5] start_thread+0xc5
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x00007f48d9132800 JavaThread "DBAdapter-DB-Pool connection adder" daemon [_thread_new, id=68714, stack(0x00007f4035156000,0x00007f4035257000)]
=>0x00007f48d9131800 JavaThread "DBAdapter-DB-Pool connection closer" daemon [_thread_new, id=68713, stack(0x00007f403babf000,0x00007f403bbc0000)]
0x00007f4914add000 JavaThread "AS40
OS:Oracle Linux Server release 7.9
uname:Linux 3.10.0-1160.2.2.el7.x86_64 #1 SMP Thu Oct 22 09:10:02 PDT 2020 x86_64
libc:glibc 2.17 NPTL 2.17
rlimit: STACK 8192k, CORE 0k, NPROC 160274, NOFILE 65535, AS infinity
load average:4.07 5.27 5.04
/proc/meminfo:
MemTotal: 41053988 kB
MemFree: 353240 kB
MemAvailable: 18211824 kB
Buffers: 0 kB
Cached: 17371124 kB
SwapCached: 30176 kB
Active: 27419984 kB
Inactive: 9387136 kB
Active(anon): 17764840 kB
Inactive(anon): 1767940 kB
Active(file): 9655144 kB
Inactive(file): 7619196 kB
Unevictable: 12380 kB
Mlocked: 12380 kB
SwapTotal: 4194300 kB
SwapFree: 3862524 kB
Dirty: 18944 kB
Writeback: 0 kB
AnonPages: 19420940 kB
Mapped: 76264 kB
Shmem: 85784 kB
Slab: 1544296 kB
SReclaimable: 1019240 kB
SUnreclaim: 525056 kB
KernelStack: 757520 kB
PageTables: 138316 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 24721292 kB
Committed_AS: 65627724 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 296108 kB
VmallocChunk: 34359310332 kB
Percpu: 28672 kB
HardwareCorrupted: 0 kB
AnonHugePages: 12632064 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 341824 kB
DirectMap2M: 8046592 kB
DirectMap1G: 35651584 kB
What is the idle solution to resolve this problem?
