Error in springboot using undertow server UT005023

Question

I have a spring boot app with a servlet registered with TCompactProtocol factory and periodically I get this error saying

2016-03-03 19:04:04,478 [XNIO-3 task-20] ERROR i.u.request - UT005023: Exception handling request to /datastream
javax.servlet.ServletException: org.apache.thrift.transport.TTransportException
    at org.apache.thrift.server.TServlet.doPost(TServlet.java:86) ~[libthrift-0.9.2.jar:0.9.2]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) ~[javax.servlet-api-3.1.0.jar:3.1.0]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
    at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:101) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:265) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:200) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:193) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:106) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at org.springframework.web.servlet.view.InternalResourceView.renderMergedOutputModel(InternalResourceView.java:168) ~[spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:303) ~[spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1243) ~[spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1027) ~[spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:971) ~[spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893) ~[spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:969) ~[spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:871) ~[spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) ~[javax.servlet-api-3.1.0.jar:3.1.0]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:845) ~[spring-webmvc-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
    at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:237) ~[spring-boot-actuator-1.3.2.RELEASE.jar:1.3.2.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:111) ~[spring-boot-actuator-1.3.2.RELEASE.jar:1.3.2.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:87) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:121) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:103) ~[spring-boot-actuator-1.3.2.RELEASE.jar:1.3.2.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) ~[undertow-core-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) ~[undertow-core-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) ~[undertow-core-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174) ~[undertow-servlet-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202) ~[undertow-core-1.3.14.Final.jar:1.3.14.Final]
    at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793) [undertow-core-1.3.14.Final.jar:1.3.14.Final]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_71]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_71]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_71]
Caused by: org.apache.thrift.transport.TTransportException: null
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[libthrift-0.9.2.jar:0.9.2]
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[libthrift-0.9.2.jar:0.9.2]
    at org.apache.thrift.protocol.TCompactProtocol.readByte(TCompactProtocol.java:634) ~[libthrift-0.9.2.jar:0.9.2]
    at org.apache.thrift.protocol.TCompactProtocol.readMessageBegin(TCompactProtocol.java:501) ~[libthrift-0.9.2.jar:0.9.2]
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) ~[libthrift-0.9.2.jar:0.9.2]
    at org.apache.thrift.server.TServlet.doPost(TServlet.java:83) ~[libthrift-0.9.2.jar:0.9.2]
    ... 97 common frames omitted

This error is random and doesn't happen every time. Anybody else noticed this issue?

I am using lib thrift 0.9.2, which I can upgrade to 0.9.3 but not sure if this will fix the issue.

The error is specifically at this point. org.apache.thrift.transport.TTransportException: null at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[libthrift-0.9.2.jar:0.9.2] — vijay chaudhary, Mar 04 '16 at 12:32

JensG · Answer 1 · 2016-03-05T00:11:57.930

0

Caused by: org.apache.thrift.transport.TTransportException: null
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
    at org.apache.thrift.protocol.TCompactProtocol.readByte(TCompactProtocol.java:634)
    at org.apache.thrift.protocol.TCompactProtocol.readMessageBegin(TCompactProtocol.java:501)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)

Looks very much like a normal client disconnect.

The server read fails in that case, causing a TTransportException. That's normal Thrift behaviour and usually an expected case, since clients are allowed to drop their end of the connection whenever they want. The problem is, the only thing that the server sees is a broken connection, but usually there is next to no chance to get the reason behind it. Hence the TTransportException.

So I'd say don't panic.

edited Mar 05 '16 at 00:11

answered Mar 04 '16 at 21:31

JensG

13,148
4
45
55

Thanks @JensG. My concern is if the connection is dropped in the middle of the operation, that would mean disruption in user experience. May be tuning the thrift client timeout would help? – vijay chaudhary Mar 04 '16 at 22:10
At least your trace shows a pending call to `readMessageBegin()`. That's the typical situation when the server is waiting for a new message (aka. service method call) coming in and the client hanging up. There is no action going on at that point. Any previous calls have been completed, and no new call has been started. – JensG Mar 05 '16 at 00:10
Unless you have a specific reason to suspect an *unexpectedly (!)* dropped connection due to timeouts, I'd not try to "tune" anything that does not even look like an problem or error so far. – JensG Mar 05 '16 at 00:14

Error in springboot using undertow server UT005023

1 Answers1