0

Problem - Memory Utilization keeps growing in my program when parsing an XML string.

I have an XML stream coming in live on a TCP port.
I use Netty to pull the xml data out and then I use XStream to deserialize/unmarshal the XML String into Java Object tree.

When I monitor memory consumption using netbeans profile, I see the HEAP growing over time and then finally JVM throws OutOfMemory exception. I traced the call Stack and I've attached a screenshot of one my tests in action.

The memory consumption seems to be happening when I deserialize/unmarshal the XML String into Java Objects.

I've tried different parsers within XStream to do the parsing - XPP3,KXML,STAX. I've even tried JAXB instead of XStream to unmarshal. No matter what parser I use, the problem persists.

I've tried all these different parsers so far but I have the same issue. 

  xstream1 = new XStream(new KXml2Driver());
  xstream2 = new XStream(new StaxDriver());
  xstream3 = new XStream(new JDomDriver());
  xstream4 = new XStream(new Xpp3Driver());

Like I mentioned,I've even tried JAXB to unmarshal instead of XStream...still same issue.

If you look at the attached image, its this Arrays.copyOfRange call right under the char[] that shows up.... No matter which parser it is, this call always shows up at the top of the trace.

I'm completely lost on how to fix or approach this problem

PLS NOTE - I'm not reading XML from a file. I get a live stream of data containing small XML chunks. I extract each chunk to convert it into Java objects for further processing

Thanks A Allocation Stack Trace from Memory Consumption data pulled using Netbeans Profiler

FatherFigure
  • 1,135
  • 1
  • 16
  • 39

2 Answers2

1

Well, on the evidence you've shown us, the simplest explanation is that your JVM's heap is too small. Try adding an "-Xmx" option as described in the manual entry for the java command.

If that doesn't work, then you need to take a deeper look at what your application is doing:

  • Is there an upper bound on the size of these "small" XML chunks? Could you be getting a chunk that is bigger than you have allowed for?

  • Is your application's processing keeping stuff from each chunk in a long lived in-memory data structure? Can you place an upper bound on the size of that data structure? Perhaps by throwing stuff out? (Or by using "weak references" so that the GC will throw them out?)

  • Is your application leaking memory?

Stephen C
  • 698,415
  • 94
  • 811
  • 1,216
  • A) No upper bound B) I dont keep anything intentionally...unless there's a bug. C) This is what I was trying to figureout and things are just pointing out at this parsing area I've mentioned, where the memory seems to be happening. – FatherFigure Jul 08 '12 at 06:50
  • If you are not retaining any data, then it is hard to see how you are leaking memory, in the XML parsing or anything else. But either way, there are ways to track down storage leaks. For example: http://www.oracle.com/technetwork/java/javase/memleaks-137499.html#gbywf – Stephen C Jul 08 '12 at 06:57
  • My application did have a memory leak - works fine after fixing it. – FatherFigure Jul 13 '12 at 02:16
1

Answer above is solid. I would only add that an object graph in memory can be expensive. If I understand your description, you have a stream of XML which contains a number of small XML chunks? SAX parsing might be your answer, along with a pipeline approach. As the SAX parser finds each chunk of work, it passes it to a downstream process without having to pull all the objects into memory.

ipaul
  • 374
  • 1
  • 10
  • If we assume that the chunks are indeed small and have a bounded size, then it is not a stretch to assume that a corresponding DOM will be small(-ish) and bounded. In that scenario, converting to a SAX parser is a lot of work ... compared with just increasing the heap size. – Stephen C Jul 08 '12 at 06:49
  • Guys - As mentioned in my post, I've already tried, STAX,XPP3,KXML2 and JDOM....All give me same issues... I want to believe its a leak in my own code, but the stack trace I see in profiler points out somewhere completely different. I'm having a hard time mapping it back to my own code. I'm currently switching to use another solution currently - Abandon XML and just do serialize/deserialize my Java Data Object between my client and server. If that also gives me a memory leak, then I could possibly try and trace the memory leak in my own code – FatherFigure Jul 08 '12 at 06:54
  • I think you are probably misinterpreting the profiler output. The fact that the failed allocation occurs in the parser does NOT mean that the parser is leaking memory. – Stephen C Jul 08 '12 at 07:03