0

We have an apache servicemix instance (version 3.3.1) in Production which runs our bpel flows (using apache ode 1.3.5) and some camel code ( for the routing ). The issue is that, the used heap space of the servicemix process keeps increasing. Eventually it runs out of space and crashes. As a result we have to keep restarting the process every 7-8 days. (which is very very annoying)

The present jvm memory configurations for the process are as follows
-Xms512M -Xmx2048M -XX:PermSize=512m -XX:MaxPermSize=1024m

We have another servicemix instance with the same memory configurations, but running under slightly lesser load which runs for about 20-22 days before it exceeds the allocated heap space. Obviously, the lesser load on this one helps it in it's extended run.

My questions

  1. Has anyone experienced the same kind of issue with the above mentioned version of apache servicemix ?
    (At an initial level i want to identify whether it is a container related leak or an application related issue )

  2. How do you go about solving this problem ? Is there a methodology that i can apply for finding out the issue ? If so, can anyone list out the steps involved in the same ?
    (Memory Leak resolution articles available on the net seems to emphasize more on the theory that causes memory leaks than on the steps that should be adopted for solving it)

Need your thoughts, suggestions and advice on this.

Thanks,
Arun Jolly

рüффп
  • 5,172
  • 34
  • 67
  • 113
Arun Jolly
  • 342
  • 1
  • 3
  • 12

2 Answers2

1

Usually when we faced this issue, we generate a Heap dump file, A heap dump is a snapshot of the memory of a Java process at a certain point of time. There are different formats for persisting this data, and depending on the format it may contain different pieces of information, but in general the snapshot contains information about the java objects and classes in the heap at the moment the snapshot was triggered.

There many way to generate a heap dump file, but in you case you could add this parameters to automaticlly generate the heap dump file when OutOfMemoryError occurred:

-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=[HeapDirPath]

So this file will allow you to figure out what are the objects that filled you whole space, then you will figure out easily the code responsible for that memory leak.

You can use Eclipse memory Analyzer to analyze this kinds of dump files.

Salah
  • 8,567
  • 3
  • 26
  • 43
1

In my experience, that type of memory problem with ServiceMix 3 is often an indication of a problem with the MEP handling of one of the components or endpoints in your flows. Some of the JBI components, endpoints and services keep a list of pending exchanges, so if some of those message exchange patterns fail to get terminated correctly, those exchanges will never be removed from the list.

The best way to troubleshoot this, would be by taking a heap dump (e.g. using jmap) and then looking at the MessageExchange implementation instances in there. You'll probably find a bunch of very similar MessageExchanges that are being kept in memory. Once you have those, you can look at the exchange properties to figure out which endpoint/component is causing the problem.

There are also a few issues that were fixed in later ServiceMix 3.x releases that could be the cause of this. Be sure to give things a go on the 3.4.1 version or check the release notes for those more recent versions.

gertv
  • 276
  • 1
  • 2
  • Thank you Gert. Can you be a bit more specific about the MEP implementation instances that i should look for in the heap dump ? The components that we use are - embedded **activemq**, **camel** component for routing from the queue to the bpel flow, **apache ode** component for running the flows and the **http provider** component for the flows to call the web services hosted outside the servicemix environment, – Arun Jolly Mar 27 '14 at 06:42
  • There's a specific MessageExchange implementation class for each of the supported MEPs. I'd look out for `org.apache.servicemix.jbi.messaging.InOnlyImpl/InOutImpl/InOptionalOutImpl/RobustInOnlyImpl` instances. Alternatively, you can also look at the underlying value object for all of these, which is called `org.apache.servicemix.jbi.messaging.ExchangePacket` – gertv Mar 27 '14 at 14:22
  • Hmm..that's very good information gert. Just being curious ( i know that we are moving beyond the scope of this question)...i've extensively tested our setup, and we've never missed any of the requests that we've pushed into servicemix ( ie all requests successfully complete the entire flow)..how then could there be mep implementations holding on to exchange objects ? Are you indicating that even after the messages are delivered, components or endpoints might still continue to hold on to exchange objects ? – Arun Jolly Mar 27 '14 at 14:53
  • All of the MEPs have to end with a DONE or ERROR message. For example: an InOnly exchange is first sent to the target endpoint in an ACTIVE state - after it's handled, that endpoint is responsible for sending back a DONE message. If the target endpoint doesn't reply with that DONE state, it may look like everything happened correctly from the outside while the previous endpoint is still waiting for the DONE message to arrive. That being said, a lot of the time, the problem is in the JBI component instead of in your own code, so you definitely want to give that 3.4.1 release a go, I think. – gertv Mar 27 '14 at 15:24
  • Yep..I've downloaded the 3.4.1 version and we've started testing our applications in it. But before i migrate my container from 3.3.1 to 3.4.1, i need to address my product stakeholders with the exact issue that is hogging us now and convince them (with data) as to why the migration is necessary. Thanks a lot gert, your inputs have been very helpful, will get back to this forum with the results. – Arun Jolly Mar 27 '14 at 15:37