3

I have a memory leak in my Jetty webapp (9.3.6) (java.lang.OutOfMemoryError: Java heap space error).

A quick note for the moderators, this is NOT a PermGen problem which seems to have documented solutions. I tried a number of SO searches and this problem did not seem to be asked about in any of the related questions.

It is a websocket application, and I am profiling it with JVisualVM, and I see that there are a large number of Strings that get allocated that don't ever get removed, and I have no idea how or what to do with that problem. In addition, I see both org.eclipse.jetty.websocket.server.WebSocketServerFactory#1 and org.eclipse.jetty.webapp.WebAppClassLoader#1 grow steadily (rapidly?) in size. To fix that problem (or so I thought) I read the memory management page in Jetty here and added the AppContextLeakPreventer to my ${jetty.home}/etc/jetty.xml and that is not helping either (discovered that my redoing my heap dump).

What's the correct way to deal with constantly growing heap in a Jetty webapp? At the moment, I am pre-production and so I am not hot-deploying the server. I have also increased the heap size to -Xmx256M (it is a 1GB RAM AWS instance, so not much elbow room here).

My client applications are at the moment few in number, and I am primarily observing constantly growing memory profile using JVisualVM.

Any input is appreciated. Any pointers as to figuring out exactly what classes are causing this is also appreciated. I currently got the aforementioned information from the heap dump in the "Biggest objects" section of the summary/inspect tab.

(edit, 02/15)

I am adding some testing notes after realizing that this is likely a bug in Jetty 9.3.6 (see http://dev.eclipse.org/mhonarc/lists/jetty-users/msg06666.html). I'd like to add that my current testing environment for this profiling work is firing up Java client sessions in very quick succession using a shell script (i.e., not describing a production environment, but this is clearly an issue for production environments as well). There are only three shells, but they are firing a brand new client session into the websocket server every 5 or so seconds, continuously (but they close the connection before they do it over again).

Sonny
  • 2,103
  • 1
  • 26
  • 34
  • It appears like I have run into this issue raised in the Jetty users mailing list: `http://dev.eclipse.org/mhonarc/lists/jetty-users/msg06530.html`. The bug was raised for 9.3.5, and I am using 9.3.6, and I guess I have to check if the fix is in. I can confirm that the Session objects and the other `WebSocketServerFactory` objects are still around, causing a lot of heap usage. – Sonny Feb 15 '16 at 14:13
  • OK, this is indeed a bug (see Joakim Erdfelt's message at http://dev.eclipse.org/mhonarc/lists/jetty-users/msg06666.html). The issue is that the Session objects are not cleaned up correctly in the JSR2356 implementation of WebSockets in Jetty 9.3.6, and in my application (clarifications added to OP), the problem gets very bad very quickly. Awaiting a fix. – Sonny Feb 15 '16 at 14:19

1 Answers1

3

I confirm that moving to Jetty 9.3.7 removed this issue; I believe the heap space issue I faced has to do with the bug (and the fix) described in here.

Specifically, there was a bug in Jetty 9.3.6 (and, presumably, Jetty 9.3.5 as well), that did not clean up JSR356 (client) Session objects correctly on the Jetty server, resulting in a memory leak for applications whose clients reconnect often (or there are too many of them).

Hope this helps someone.

Sonny
  • 2,103
  • 1
  • 26
  • 34