How can i solve the problem Axon server Stream Error

Question

I am implementing CQRS with Axon + Spring Boot. My command handles a long computation which results on a large aggregate. The problem is after some minutes the Axon server seems to cancel the command and after all an error is thrown in axon server, please see the log below:

LOGS FROM AXON SERVER:

io.netty.handler.codec.http2.Http2Exception$StreamException: Received DATA frame for an unknown stream 11 at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:129) ~[netty-codec-http2-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.shouldIgnoreHeadersOrDataFrame(DefaultHttp2ConnectionDecoder.java:531) ~[netty-codec-http2-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:183) ~[netty-codec-http2-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48) ~[netty-codec-http2-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:422) ~[netty-codec-http2-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:251) ~[netty-codec-http2-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160) ~[netty-codec-http2-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41) ~[netty-codec-http2-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118) ~[netty-codec-http2-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:390) ~[netty-codec-http2-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:450) [netty-codec-http2-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) [netty-codec-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) [netty-codec-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) [netty-codec-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) [netty-transport-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:617) [netty-transport-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:534) [netty-transport-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.35.Final.jar!/:4.1.35.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.35.Final.jar!/:4.1.35.Final] at java.lang.Thread.run(Unknown Source) [na:1.8.0_181]

Thats all i got, and i don't know why... I am using the latest versions of axon and axon server (4.2.1)

Answer 1

Axon Server has a timeout for commands that is set by default to 5 minutes (300_000 ms). You can increase this timeout by setting the property axoniq.axonserver.default-command-timeout. The value of the property is the timeout in milliseconds. The maximum message size is 4Mb by default. You can increase that by setting the property axoniq.axonserver.max-message-size in axonserver.properties.

Hope this solves your issue.

Marc

Answer 2

The message "Cancelling command pt.some.admtools.inhousecatalog.command.CheckoutAndParseCommand sent by 16912@POR-5CG8271235, waiting for reply from 16912@POR-5CG8271235.default" is sent when AxonServer cancels the command after 5 minutes (axoniq.axonserver.default-command-timeout). Can you check what happens in your command handler, and how long the handling takes?