I have an Axis2 1.5.5 instance running on Tomcat5.5. Tomcat connector configuration is like below.
<Connector port="8080" maxHttpHeaderSize="8192"
maxThreads="100" minSpareThreads="15" maxSpareThreads="40"
enableLookups="false" redirectPort="8443" acceptCount="100"
connectionTimeout="20000" disableUploadTimeout="true" />
Client send post request to my service endpoint deployed in Axis2, but receive read timed out just after few milliseconds.
I check this through TCP dumps.
Access logs looks like below.
http-5905-Processor28 "POST /axis2/services/my_service_endpoint HTTP/1.1" 500 4234 20003
Access logs shows that it waited for 20 seconds as per connectionTiemout attribute.
This is the stack trace received by client.
org.apache.axis2.AxisFault: Read timed out
at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
at org.apache.axis2.builder.SOAPBuilder.processDocument(SOAPBuilder.java:75)
at org.apache.axis2.transport.TransportUtils.createDocumentElement(TransportUtils.java:197)
at org.apache.axis2.transport.TransportUtils.createSOAPMessage(TransportUtils.java:145)
at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:168)
at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:144)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:879)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:783)
at org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:813)
at org.apache.coyote.http11.filters.ChunkedInputFilter.readBytes(ChunkedInputFilter.java:226)
at org.apache.coyote.http11.filters.ChunkedInputFilter.parseChunkHeader(ChunkedInputFilter.java:249)
at org.apache.coyote.http11.filters.ChunkedInputFilter.doRead(ChunkedInputFilter.java:130)
at org.apache.coyote.http11.InternalInputBuffer.doRead(InternalInputBuffer.java:748)
at org.apache.coyote.Request.doRead(Request.java:419)
at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:270)
at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:403)
at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:293)
at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:193)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
at org.apache.axiom.om.util.DetachableInputStream.read(DetachableInputStream.java:147)
at java.io.FilterInputStream.read(Unknown Source)
at java.io.PushbackInputStream.read(Unknown Source)
at org.apache.axis2.builder.BuilderUtil.getCharSetEncoding(BuilderUtil.java:294)
at org.apache.axis2.builder.SOAPBuilder.processDocument(SOAPBuilder.java:56)
... 24 more
First I thought it could be due to some network issue, but later our network admin confirmed he didn't find any issue within network.
Not sure why its getting read timed out just after few milliseconds. Retrying the same request after sometime will go through successfully. This is an intermittent behavior but frequency is very high.
I am looking for a hint to further investigate this issue. Thanks for your help.
