Spring boot microservice (jdk11) threw hystrix.stream-actuator-endpoint exception

Question

After spring boot service migration to 2.1.3.RELEASE and jdk11, hystrix stream actuator-related exceptions occur.

The exception is ERROR org.apache.catalina.core.ContainerBase.[Tomcat-1].[localhost].[/].[hystrix.stream-actuator-endpoint] - Servlet.service() for servlet [hystrix.stream-actuator-endpoint] in context with path [] threw exception[]

Spring boot version: 2.1.3.RELEASE

Java version: 11

All actuator endpoints are exposed: management.endpoints.web.exposure.include=*

Also, hystrix.stream endpoint is actually enabled and gives a stream. Nevertheless, exceptions still occur in logs

Stack traces:

java.nio.BufferOverflowException
    at java.base/java.nio.DirectByteBuffer.put(DirectByteBuffer.java:410)
    at java.base/java.nio.DirectByteBuffer.put(DirectByteBuffer.java:389)
    at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:74)
    at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:50)
    at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:466)
    at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:134)
    at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:105)
    at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:144)
    at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1223)
    at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:743)
    at org.apache.tomcat.util.net.SocketWrapperBase.flushBlocking(SocketWrapperBase.java:696)
    at org.apache.tomcat.util.net.SocketWrapperBase.flush(SocketWrapperBase.java:686)
    at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.flush(Http11OutputBuffer.java:553)
    at org.apache.coyote.http11.filters.ChunkedOutputFilter.flush(ChunkedOutputFilter.java:157)
    at org.apache.coyote.http11.Http11OutputBuffer.flush(Http11OutputBuffer.java:216)
    at org.apache.coyote.http11.Http11Processor.flush(Http11Processor.java:1149)
    at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:394)
    at org.apache.coyote.Response.action(Response.java:209)
    at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:295)
    at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:262)
    at org.apache.catalina.connector.CoyoteWriter.flush(CoyoteWriter.java:94)
    at org.apache.catalina.connector.CoyoteWriter.checkError(CoyoteWriter.java:119)
    at com.netflix.hystrix.contrib.sample.stream.HystrixSampleSseServlet.handleRequest(HystrixSampleSseServlet.java:165)
    at com.netflix.hystrix.contrib.sample.stream.HystrixSampleSseServlet.doGet(HystrixSampleSseServlet.java:74)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:200)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.base/java.lang.Thread.run(Thread.java:834)

and a similar one

java.lang.IllegalArgumentException: newPosition > limit: (3850 > 532)
    at java.base/java.nio.Buffer.createPositionException(Buffer.java:318)
    at java.base/java.nio.Buffer.position(Buffer.java:293)
    at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1086)
    at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:82)
    at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:50)
    at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:466)
    at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:134)
    at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:105)
    at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:144)
    at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1223)
    at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:743)
    at org.apache.tomcat.util.net.SocketWrapperBase.flushBlocking(SocketWrapperBase.java:696)
    at org.apache.tomcat.util.net.SocketWrapperBase.flush(SocketWrapperBase.java:686)
    at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.flush(Http11OutputBuffer.java:553)
    at org.apache.coyote.http11.filters.ChunkedOutputFilter.flush(ChunkedOutputFilter.java:157)
    at org.apache.coyote.http11.Http11OutputBuffer.flush(Http11OutputBuffer.java:216)
    at org.apache.coyote.http11.Http11Processor.flush(Http11Processor.java:1149)
    at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:394)
    at org.apache.coyote.Response.action(Response.java:209)
    at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:295)
    at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:262)
    at org.apache.catalina.connector.CoyoteWriter.flush(CoyoteWriter.java:94)
    at com.netflix.hystrix.contrib.sample.stream.HystrixSampleSseServlet.handleRequest(HystrixSampleSseServlet.java:168)
    at com.netflix.hystrix.contrib.sample.stream.HystrixSampleSseServlet.doGet(HystrixSampleSseServlet.java:74)

I suppose it should be ok with default settings. It's a simple scenario and should work without customization — yevtsy, Apr 16 '19 at 08:53
Have you resolve this problem? I've got same problem but no result from google. — Se ven, Jun 28 '19 at 09:40
From official docs: To enable the Hystrix metrics stream, include a dependency on spring-boot-starter-actuator and set management.endpoints.web.exposure.include: hystrix.stream. Doing so exposes the /actuator/hystrix.stream as a management endpoint. I think it should be enough! Reference https://cloud.spring.io/spring-cloud-netflix/reference/html/#hystrix-metrics-stream — Roberto Manfreda, Dec 04 '19 at 09:25
Can you provide us with a minimum configuration sample to reproduce the problem? Some code here or a link over GitHub or elsewhere? — Roberto Manfreda, Dec 04 '19 at 09:29
@RobertoManfreda yes, all required steps are done. That's why we are receiving hystrix.stream events. Unfortunately, some errors occur from time to time. I can't get a link to the configuration but you can reproduce it based on official docs setup + jdk11 + spring boot 2.1.3.RELEASE (also can try to bump to latest version) — yevtsy, Dec 04 '19 at 09:42
Perfect, so I'll try to reproduce the problem... I'll let you know — Roberto Manfreda, Dec 04 '19 at 09:43
Looks like this happens when the stream data is greater than writer buffer size. Can you try to increase the value by setting `socket.appWriteBufSize` to a value more than `8192`? — Sunil Dabburi, Dec 10 '19 at 16:06

Roberto Manfreda · Answer 1 · 2019-12-10T02:15:05.157

Running the minimal configuration I'm unable to reproduce the problem, I tried switching version from 2.1.3-RELEASE to 2.2.2-RELEASE too; no problems. So I think you should give more details here. Where is executing your application? Embedded tomcat, external tomcat, docker? Try to give more details please.

Anyway here's my test configuration.

pom.xml

 <?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.1.3.RELEASE</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.robman</groupId>
    <artifactId>demohystrix</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>demohystrix</name>
    <description>Demo project for Spring Boot</description>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>

        <java.version>11</java.version>

        <spring-cloud.version>Greenwich.SR3</spring-cloud.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
        </dependency>
    </dependencies>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>${spring-cloud.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

application.properties

management.endpoints.web.exposure.include=*

DemoHystrixApplication.java

@SpringBootApplication
@EnableHystrix
@RestController
public class DemoHystrixApplication {

    public static void main(String[] args) {
        SpringApplication.run(DemoHystrixApplication.class, args);
    }

    @RequestMapping(value = "/")
    @HystrixCommand(fallbackMethod = "fallback", commandProperties = {
            @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "1000")
    })

    public String hello() throws InterruptedException {
        Thread.sleep(3000);
        return "HELLO Hystrix";
    }

    private String fallback() {
        return "Fallback";
    }

}

In case you are using Eureka server:
Official docs says

The JAXB modules which the Eureka server depends upon were removed in JDK 11. If you intend to use JDK 11 when running a Eureka server you must include these dependencies in your POM or Gradle file.
<dependency>
    <groupId>org.glassfish.jaxb</groupId>
    <artifactId>jaxb-runtime</artifactId>
</dependency>

I'm also unable to always reproduce this error. It appears not often. But after removing hystrix-stream-actuator, no exceptions for 2 weeks happened. — Max, Dec 10 '19 at 08:30

Spring boot microservice (jdk11) threw hystrix.stream-actuator-endpoint exception

1 Answers1