Unexpected error [32] writing data to the APR/native socket

Question

Just upgraded java springboot project to:

springboot 2.6.6
tomcat 9
jdk11
war

some pointers:

I dont use websockets in my project
I dont write any files to my server

started getting this exception:

org.apache.catalina.connector.ClientAbortException: java.io.IOException: java.io.IOException: Unexpected error [32] writing data to the APR/native socket [140,207,466,892,064] with wrapper [org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper@2bb69ce5:140207466892064].
       at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:353)
       at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:784)
       at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:689)
       at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:388)
       at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:366)
       at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96)
       at org.springframework.util.StreamUtils$NonClosingOutputStream.write(StreamUtils.java:287)
       at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2171)
       at com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1184)
       at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:915)
       at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:456)
       at org.springframework.http.converter.AbstractGenericHttpMessageConverter.write(AbstractGenericHttpMessageConverter.java:104)
       at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:290)
       at org.springframework.web.servlet.mvc.method.annotation.HttpEntityMethodProcessor.handleReturnValue(HttpEntityMethodProcessor.java:219)
       at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:78)
       at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:135)
       at org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver.doResolveHandlerMethodException(ExceptionHandlerExceptionResolver.java:428)
       at org.springframework.web.servlet.handler.AbstractHandlerMethodExceptionResolver.doResolveException(AbstractHandlerMethodExceptionResolver.java:75)
       at org.springframework.web.servlet.handler.AbstractHandlerExceptionResolver.resolveException(AbstractHandlerExceptionResolver.java:142)
       at org.springframework.web.servlet.handler.HandlerExceptionResolverComposite.resolveException(HandlerExceptionResolverComposite.java:80)
       at org.springframework.web.servlet.DispatcherServlet.processHandlerException(DispatcherServlet.java:1327)
       at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1138)
       at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1084)
       at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
       at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
       at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:655)
       at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
       at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
       at com.walmart.chp.pharmacy.txndata.cosmosingest.filter.RequestResponseFilter.doFilterInternal(RequestResponseFilter.java:28)
       at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
       at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
       at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
       at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
       at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
       at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
       at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
       at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:126)
       at org.springframework.boot.web.servlet.support.ErrorPageFilter.access$000(ErrorPageFilter.java:64)
       at org.springframework.boot.web.servlet.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:101)
       at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
       at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:119)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
       at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
       at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

The stack trace is bigger than this, please let me know how I can share more, as stackoverflow is not allowing me to paste more "code" in the question.

edit:

Caused by: java.io.IOException: java.io.IOException: Unexpected error [32] writing data to the APR/native socket [140,207,466,892,064] with wrapper [org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper@2bb69ce5:140207466892064].
       at org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper.doWriteInternal(AprEndpoint.java:2486)
       at org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper.doWrite(AprEndpoint.java:2433)
       at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:768)
       at org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:593)

Caused by: java.io.IOException: Unexpected error [32] writing data to the APR/native socket [140,207,466,892,064] with wrapper [org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper@2bb69ce5:140207466892064].
       at org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper.doWriteInternal(AprEndpoint.java:2525)
       at org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper.doWrite(AprEndpoint.java:2433)
       at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:768)
       at org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:593)
       at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:537)

Server is taking too long to respond and thus client is aborting the connection. Please check if there is anything in your spring boot code which is slowing down the server. — SRJ, Aug 26 '22 at 04:46
hi @shubham that is the issue, when I run this locally I dont see this, but when this application runs in a docker container in a k8s cluster, the exception handler catches this ex but I am not able to understand who is the caller, or what was the call made. — Udaishankar T, Sep 17 '22 at 13:40
If you're logging your requests then docker logs would show what requests are coming to this application or it may be happening at startup of the application ? btw how is app logging configured ? — SRJ, Sep 17 '22 at 14:09
hi @shubham Correct, so I am logging incoming-outgoing req-resp via oncePerRequestFilter, logging is via log4j2 and xml configuration, I am just printing it on the console. I see all the requests in the logs except for the call that is causing this exception. Also there is an istio proxy running in the app container, which logs all the calls incoming and outgoing, via istio logs the only calls failing are /prometheus(which I have exposed for metrics scrapping in the app.props), which is giving 503 UF. But is it a websocket call? — Udaishankar T, Sep 18 '22 at 15:09
Are you able to connect to your prometheus instance ? you can try to post some data directly to prometheus and validate it. — SRJ, Sep 19 '22 at 12:11

Unexpected error [32] writing data to the APR/native socket

0 Answers0