2

I'm having problems deployments a WAR in JBoss EAP 7.0.0. This war is based on an Spring application.

After deploying and removing the WAR 3 or 4 times, I get an OutOfMemoryError exception:

17:02:37,643 ERROR [stderr] (Thread-223) Exception in thread "Thread-223" java.lang.OutOfMemoryError: Java heap space
17:02:37,645 ERROR [stderr] (Thread-223)    at io.netty.util.internal.PlatformDependent.allocateUninitializedArray(PlatformDependent.java:189)
17:02:37,645 ERROR [stderr] (Thread-223)    at io.netty.buffer.PoolArena$HeapArena.newByteArray(PoolArena.java:676)
17:02:37,645 ERROR [stderr] (Thread-223)    at io.netty.buffer.PoolArena$HeapArena.newChunk(PoolArena.java:686)
17:02:37,646 ERROR [stderr] (Thread-223)    at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244)
17:02:37,646 ERROR [stderr] (Thread-223)    at io.netty.buffer.PoolArena.allocate(PoolArena.java:226)
17:02:37,646 ERROR [stderr] (Thread-223)    at io.netty.buffer.PoolArena.allocate(PoolArena.java:146)
17:02:37,646 ERROR [stderr] (Thread-223)    at io.netty.buffer.PooledByteBufAllocator.newHeapBuffer(PooledByteBufAllocator.java:307)
17:02:37,647 ERROR [stderr] (Thread-223)    at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:162)
17:02:37,647 ERROR [stderr] (Thread-223)    at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:153)
17:02:37,647 ERROR [stderr] (Thread-223)    at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:135)
17:02:37,648 ERROR [stderr] (Thread-223)    at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:80)
17:02:37,648 ERROR [stderr] (Thread-223)    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:122)
17:02:37,648 ERROR [stderr] (Thread-223)    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
17:02:37,648 ERROR [stderr] (Thread-223)    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
17:02:37,649 ERROR [stderr] (Thread-223)    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
17:02:37,649 ERROR [stderr] (Thread-223)    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
17:02:37,649 ERROR [stderr] (Thread-223)    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
17:02:37,649 ERROR [stderr] (Thread-223)    at java.lang.Thread.run(Thread.java:748)
17:04:30,709 ERROR [stderr] (MyThread_ViewsDtoGeneration) Exception in thread "MyThread_ViewsDtoGeneration" java.lang.OutOfMemoryError: GC overhead limit exceeded
17:04:30,711 ERROR [stderr] (MyThread_ViewsDtoGeneration)   at java.util.Arrays.copyOfRange(Arrays.java:3664)
17:04:30,711 ERROR [stderr] (MyThread_ViewsDtoGeneration)   at java.lang.String.<init>(String.java:207)
17:04:30,711 ERROR [stderr] (MyThread_ViewsDtoGeneration)   at java.lang.StringBuilder.toString(StringBuilder.java:407)
17:04:30,712 ERROR [stderr] (MyThread_ViewsDtoGeneration)   at org.hibernate.dialect.Dialect.applyLocksToSql(Dialect.java:1439)
17:04:30,712 ERROR [stderr] (MyThread_ViewsDtoGeneration)   at org.hibernate.loader.plan.exec.query.internal.SelectStatementBuilder.toStatementString(SelectStatementBuilder.java:222)
17:04:30,712 ERROR [stderr] (MyThread_ViewsDtoGeneration)   at org.hibernate.loader.plan.exec.internal.AbstractLoadQueryDetails.generate(AbstractLoadQueryDetails.java:183)
17:04:30,712 ERROR [stderr] (MyThread_ViewsDtoGeneration)   at org.hibernate.loader.plan.exec.internal.EntityLoadQueryDetails.<init>(EntityLoadQueryDetails.java:90)
17:04:30,712 ERROR [stderr] (MyThread_ViewsDtoGeneration)   at org.hibernate.loader.plan.exec.internal.BatchingLoadQueryDetailsFactory.makeEntityLoadQueryDetails(BatchingLoadQueryDetailsFactory.java:61)
...

I used VisualVM to analyze what's happening in the server, and I realized that the number loaded Classes was increased in each deployment (that's OK) but it doesn't decrease when I remove the application.

Also the number of threads seems to be increased each time, but never decreased the same amount of threads created during the deployment.

I don't really know what is happening. I test the same application in a Tomcat 8.5.24, and I never get this error, and that makes me think that the error is not in my application, but who knows...

Any idea? Thanks!

James R. Perkins
  • 16,800
  • 44
  • 60
Marc Gil Sendra
  • 819
  • 2
  • 9
  • 22
  • I've been using JBoss for years and this happens after numerous deploy and undeploy operations. You basically want to bump up the JBoss memory allocation considerably if you have'net already. Note, this likely wont help permanently, e.g. eventually you'll still get the problem but it will occur less frequently. Let me know if you want to try it and I'll find my notes. – JGlass May 31 '18 at 15:49
  • @JGlass do you mean that increasing the Xms and Xmx parameters is enough to solve it? Why it doesn't occur in Tomcat, using the same memory dedication? Supposedly JBoss manages better the memory, but in this case it seems that it doesn't. Thanks! – Marc Gil Sendra May 31 '18 at 20:18
  • It is! You can try it out and either update this question with your answer of how you solved it or I will :-) Just make it solves it first. Always has helped me - been an issue in Jboss for about the last 12 years :-/ – JGlass May 31 '18 at 21:36
  • I tried assigning 4G as Xms and Xmx, and it fails after 5 deployings of the application... :( – Marc Gil Sendra Jun 01 '18 at 11:14
  • This is what I put on my machine when it started having issues - I was deploying new code to it nightly - granted the box has like 16GB of ram. ` -Xms6144m -Xmx6144m -XX:MaxPermSize=3072m` and I also have (but may be a JBoss default) `-Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000` also, how big is your WAR file? – JGlass Jun 01 '18 at 14:41
  • I'll try it next Monday and tell you the result. The war is "only" 115MB – Marc Gil Sendra Jun 01 '18 at 22:49
  • That's a rather large WAR - if you have a lot of third party libraries - you might consider taking them out of the deployment and instead add them as modules. – JGlass Jun 04 '18 at 14:33
  • - If the new generation size is explicitly defined with JVM options, decrease the size or remove the relevant JVM options entirely to unconstrain the JVM and provide more space in the old generation for long lived objects. - If there is unintended object retention, typically code and/or configuration changes are needed. - If the retention looks normal, and it is a load issue, the Java heap (-Xmx). would need to be increased. – Anup Dey Jun 05 '18 at 14:51
  • @AnupDey I don't understand what you mean with with the "new generation size..." and "defining it with JVM options". – Marc Gil Sendra Jun 05 '18 at 19:39
  • This error is thrown by the throughput old collector (serial or parallel) if more than 98% of the total time is spent doing garbage collection and less than 2% of the heap is recovered. It is intended to prevent applications from running for an extended period of time while making little or no progress reclaiming objects (e.g. when the heap is too small, there is a memory leak, or the old generation is disproportionately small compared to the new generation). It is a throttle to prevent the JVM from swamping the environment it is running in. – Anup Dey Jun 06 '18 at 11:18
  • Its necessary to increase the server the Jboss JVM's maximum heap memory setting. – Anup Dey Jun 06 '18 at 11:20
  • But increasing the maximum heap memory Xmx doesn't solve the problem, only postpones it, isn't it? – Marc Gil Sendra Jun 06 '18 at 15:06
  • The GC throws this exception when too much time is spent in garbage collection for too little return, eg. 98% of CPU time is spent on GC and less than 2% of heap is recovered. This feature is designed to prevent applications from running for an extended period of time while making little or no progress because the heap is too small. Can you can turn this off with the command line option -XX:-UseGCOverheadLimit and check. More info see: http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#par_gc.oom – Anup Dey Jun 07 '18 at 11:08
  • After a garbage collection, if the Java process is spending more than approximately 98% of its time doing garbage collection and if it is recovering less than 2% of the heap and has been doing so far the last 5 (compile time constant) consecutive garbage collections, then a java.lang.OutOfMemoryError is thrown – Anup Dey Jun 07 '18 at 11:11
  • 1.Increase the heap size if current heap is not enough. 2.If you still get this error after increasing heap memory, use memory profiling tools like MAT ( Memory analyzer tool)[http://www.eclipse.org/mat/ ], Visual VM[http://www.oracle.com/splash/java.net/maintenance/index.html ] etc and fix memory leaks. 3.Upgrade JDK version to latest version ( 1.8.x) or at least 1.7.x and use G1GC algorithm. . The throughput goal for the G1 GC is 90 percent application time and 10 percent garbage collection time – Anup Dey Jun 07 '18 at 11:12
  • 4.Apart from setting heap memory with -Xms1g -Xmx2g , try -XX:+UseG1GC -XX:G1HeapRegionSize=n -XX:MaxGCPauseMillis=m -XX:ParallelGCThreads=n -XX:ConcGCThreads=n – Anup Dey Jun 07 '18 at 11:14
  • Thanks @AnupDey. I'll try some comments that all of you said. But I think that at the end is a JBoss problem. Why? Because Tomcat has not this issues. Proved by me in same conditions of JBoss. – Marc Gil Sendra Jun 07 '18 at 19:21

0 Answers0