I have a springboot based microservice that reads an XML from a SOAP endpoint, parses it and forwards the parsed response to requesting clients.
Below is the version of springboot/spring/tomcat used:
<spring.boot.version>1.5.4.RELEASE</spring.boot.version>
<spring.version>4.3.9.RELEASE</spring.version>
Tomcat 8.5.21.0The service is functionally working fine and is able to handle load as well without much issues. But, when I look at the flight recording of the service, I see a lot of (~ million per min) exceptions of the type shown below:
Stack Trace Count Percentage(%) Total
java.lang.Throwable.<init>(String) 1,532,051 95.618 0
java.lang.Exception.<init>(String) 1,532,051 95.618 0
java.io.IOException.<init>(String) 1,503,857 93.858 0
java.io.FileNotFoundException.<init>(String) 1,503,782 93.854 0
org.springframework.boot.loader.jar.JarURLConnection.throwFileNotFound(Object, JarFile) 1,503,782 93.854 0
org.springframework.boot.loader.jar.JarURLConnection.connect() 1,503,782 93.854 0
org.springframework.boot.loader.jar.JarURLConnection.getInputStream() 1,503,782 93.854 0
sun.misc.URLClassPath$Loader.findResource(String, boolean) 1,503,782 93.854 0
sun.misc.URLClassPath$1.next() 1,503,782 93.854 0
sun.misc.URLClassPath$1.hasMoreElements() 1,503,782 93.854 0
java.net.URLClassLoader$3$1.run() 1,503,782 93.854 0
java.net.URLClassLoader$3$1.run() 1,503,782 93.854 0
java.security.AccessController.doPrivileged(PrivilegedAction, AccessControlContext) 1,503,782 93.854 0
java.net.URLClassLoader$3.next() 1,503,782 93.854 0
java.net.URLClassLoader$3.hasMoreElements() 1,503,782 93.854 0
sun.misc.CompoundEnumeration.next() 1,503,782 93.854 0
sun.misc.CompoundEnumeration.hasMoreElements() 1,503,782 93.854 0
sun.misc.CompoundEnumeration.next() 1,503,782 93.854 0
sun.misc.CompoundEnumeration.hasMoreElements() 1,503,782 93.854 0
java.util.ServiceLoader$LazyIterator.hasNextService() 1,503,782 93.854 0
java.util.ServiceLoader$LazyIterator.hasNext() 1,503,782 93.854 0
java.util.ServiceLoader$1.hasNext() 1,503,782 93.854 0
javax.xml.transform.FactoryFinder$1.run() 1,046,435 65.31 0
java.security.AccessController.doPrivileged(PrivilegedAction) 1,046,435 65.31 0
javax.xml.transform.FactoryFinder.findServiceProvider(Class) 1,046,435 65.31 0
javax.xml.transform.FactoryFinder.find(Class, String) 1,046,435 65.31 0
javax.xml.transform.TransformerFactory.newInstance() 1,046,435 65.31 0
com.sun.xml.messaging.saaj.util.transform.EfficientStreamingTransformer.<init>() 1,046,435 65.31 0
com.sun.xml.messaging.saaj.util.transform.EfficientStreamingTransformer.newTransformer() 1,046,435 65.31 0
com.sun.xml.messaging.saaj.soap.impl.EnvelopeImpl.output(OutputStream) 589,228 36.775 0
com.sun.xml.messaging.saaj.soap.impl.EnvelopeImpl.output(OutputStream, boolean) 589,228 36.775 0
com.sun.xml.messaging.saaj.soap.SOAPPartImpl.getContentAsStream() 589,228 36.775 0
com.sun.xml.messaging.saaj.soap.MessageImpl.getHeaderBytes() 589,228 36.775 0
com.sun.xml.messaging.saaj.soap.MessageImpl.saveChanges() 589,228 36.775 0
org.springframework.ws.soap.saaj.SaajSoapMessage.writeTo(OutputStream) 589,228 36.775 0
org.springframework.ws.transport.AbstractWebServiceConnection.send(WebServiceMessage) 589,228 36.775 0
org.springframework.ws.client.core.WebServiceTemplate.sendRequest(WebServiceConnection, WebServiceMessage) 589,228 36.775 0
org.springframework.ws.client.core.WebServiceTemplate.doSendAndReceive(MessageContext, WebServiceConnection, WebServiceMessageCallback, WebServiceMessageExtractor) 589,228 36.775 0
org.springframework.ws.client.core.WebServiceTemplate.sendAndReceive(String, WebServiceMessageCallback, WebServiceMessageExtractor) 589,228 36.775 0
org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(String, Object, WebServiceMessageCallback) 589,228 36.775 0
org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(Object, WebServiceMessageCallback) 589,228 36.775 0
org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(Object) 589,228 36.775 0
com.hotels.booking.discos.loyalty.client.LoyaltyWebServiceGateway.sendAndReceive(AbstractLoyaltyRequest) 589,228 36.775 0
com.hotels.booking.discos.loyalty.client.LoyaltyWebServiceGateway.retrieveRewardAdjustments(LoyaltyRewardAdjustmentsRetrievalRequest) 217,970 13.604 0
sun.reflect.GeneratedMethodAccessor334.invoke(Object, Object[]) 217,970 13.604 0
sun.reflect.DelegatingMethodAccessorImpl.invoke(Object, Object[]) 217,970 13.604 0
java.lang.reflect.Method.invoke(Object, Object[]) 217,970 13.604 0
com.hotels.thermos.proxy.GenericProxyCircuitBreaker.execute() 217,970 13.604 0
com.hotels.thermos.AbstractCircuitBreaker.run() 217,970 13.604 0
com.hotels.thermos.HystrixCommandWrapper.run() 217,970 13.604 0
com.netflix.hystrix.HystrixCommand$1.call(Subscriber) 217,970 13.604 0
com.netflix.hystrix.HystrixCommand$1.call(Object) 217,970 13.604 0
rx.Observable$2.call(Subscriber) 217,970 13.604 0
rx.Observable$2.call(Object) 217,970 13.604 0
rx.Observable$2.call(Subscriber) 217,970 13.604 0
rx.Observable$2.call(Object) 217,970 13.604 0
rx.Observable$2.call(Subscriber) 217,970 13.604 0
rx.Observable$2.call(Object) 217,970 13.604 0The specific exceptions seems to be:
Message: JAR entry META-INF /services/* not found in /<service.jar>
The number of exceptions puts a lot of memory pressure on the service, limiting the performance/scalability of the service
I tried looking for solutions to this and found a few similar issues. Most resources point to embedded tomcat as the issue. But I could not find the reason or solution for this specific issue anywhere.
Any help understanding/resolving this issue will be greatly appreciated!
