Java OutOfMemory exception: mmap failed for CEN and END part of zip file

Question

I run my app on production env (Centos7 x64, oracle jre 1.7_05,64bit VM tomcat 6.0.43) with JVM arguments

 -Xmx4123M -Xms4123M -Xmn800M -XX:PermSize=500M -XX:MaxPermSize=500M -Xss256K -XX:+DisableExplicitGC -XX:SurvivorRatio=1 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSClassUnloadingEnabled -XX:LargePageSizeInBytes=128M -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0

Looking my monitor, everthing is ok (heap and non-heap memory used for 10%) and i have no idea where is the problem.this problem is appear every ten days or so.

stderr.log

mmap failed for CEN and END part of zip file
mmap failed for CEN and END part of zip file
mmap failed for CEN and END part of zip file
mmap failed for CEN and END part of zip file
mmap failed for CEN and END part of zip file
mmap failed for CEN and END part of zip file
mmap failed for CEN and END part of zip file
mmap failed for CEN and END part of zip file
mmap failed for CEN and END part of zip file
mmap failed for CEN and END part of zip file
mmap failed for CEN and END part of zip file
mmap failed for CEN and END part of zip file
mmap failed for CEN and END part of zip file
mmap failed for CEN and END part of zip file
mmap failed for CEN and END part of zip file

stdout.log

Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled.
Java HotSpot(TM) 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=
Code Cache  [0x00007fa9240a4000, 0x00007fa924844000, 0x00007fa9270a4000)
 total_blobs=2188 nmethods=1693 adapters=452 free_code_cache=41447Kb largest_free_block=42339968
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled.
Java HotSpot(TM) 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=
Code Cache  [0x00007fa9240a4000, 0x00007fa924844000, 0x00007fa9270a4000)
 total_blobs=2190 nmethods=1694 adapters=452 free_code_cache=41446Kb largest_free_block=42339968
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard pages failed.

tomcat localhost.log

十一月 30, 2016 11:18:04 上午 org.apache.catalina.core.StandardWrapperValve invoke
严重: Servlet.service() for servlet springMVC threw exception
java.lang.OutOfMemoryError
    at java.util.zip.ZipFile.open(Native Method)
    at java.util.zip.ZipFile.<init>(ZipFile.java:214)
    at java.util.zip.ZipFile.<init>(ZipFile.java:144)
    at java.util.jar.JarFile.<init>(JarFile.java:152)
    at java.util.jar.JarFile.<init>(JarFile.java:89)
    at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93)
    at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
    at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:88)
    at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
    at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:150)
    at freemarker.cache.URLTemplateSource.close(URLTemplateSource.java:139)
    at freemarker.cache.URLTemplateLoader.closeTemplateSource(URLTemplateLoader.java:106)
    at freemarker.cache.MultiTemplateLoader$MultiSource.close(MultiTemplateLoader.java:188)
    at freemarker.cache.MultiTemplateLoader.closeTemplateSource(MultiTemplateLoader.java:142)
    at freemarker.cache.TemplateCache.getTemplate(TemplateCache.java:353)
    at freemarker.cache.TemplateCache.getTemplate(TemplateCache.java:205)
    at freemarker.template.Configuration.getTemplate(Configuration.java:740)
    at freemarker.core.Environment.getTemplateForInclusion(Environment.java:1694)
    at freemarker.core.Environment.getTemplateForImporting(Environment.java:1748)
    at freemarker.core.Environment.importLib(Environment.java:1733)
    at freemarker.template.Configuration.doAutoImportsAndIncludes(Configuration.java:1105)
    at freemarker.core.Configurable.doAutoImportsAndIncludes(Configurable.java:1271)
    at freemarker.core.Configurable.doAutoImportsAndIncludes(Configurable.java:1271)
    at freemarker.core.Environment.process(Environment.java:242)
    at freemarker.template.Template.process(Template.java:277)
    at org.springframework.web.servlet.view.freemarker.FreeMarkerView.processTemplate(FreeMarkerView.java:366)
    at org.springframework.web.servlet.view.freemarker.FreeMarkerView.doRender(FreeMarkerView.java:283)
    at org.springframework.web.servlet.view.freemarker.FreeMarkerView.renderMergedTemplateModel(FreeMarkerView.java:233)
    at org.springframework.web.servlet.view.AbstractTemplateView.renderMergedOutputModel(AbstractTemplateView.java:167)
    at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:267)
    at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1221)
    at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1005)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:952)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:870)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:863)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:643)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
    at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
    at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
    at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
    at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:620)
    at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
    at java.lang.Thread.run(Thread.java:722)

Possible duplicate of [Java OutOfMemory exception: mmap error on loading zip file](http://stackoverflow.com/questions/12815309/java-outofmemory-exception-mmap-error-on-loading-zip-file) — Robby Cornelissen, Jan 23 '17 at 02:02
But this questions did not hava a result. At least He has not specified what action he would like them to take . — si chen, Jan 23 '17 at 02:18

score 1 · Answer 1 · answered Jan 28 '17 at 21:40

1

I was having a similar issue, running this at startup resolved it for me:

sudo sysctl -w vm.max_map_count=131072

The default value on my machine was 65536, so this doubled it.

answered Jan 28 '17 at 21:40

Larry

348
1
9

The production env is same? – si chen Feb 04 '17 at 02:43
I doubt this problem is caused by jdk version, now i changed jdk version. if not, I'll try edit max_map_count – si chen Feb 04 '17 at 02:49
Yes - this was in all environments - the jdk was not the issue at all. – Larry Feb 05 '17 at 04:05
BTW, to test this, you can just run this command from a terminal, then try your app. I had to run it at startup because I'm running my app on an AWS EC2 spot instance. – Larry Feb 05 '17 at 04:07
I think ,what caused this problem? I don't understand! – si chen Feb 08 '17 at 04:00

Java OutOfMemory exception: mmap failed for CEN and END part of zip file

1 Answers1