My ActiveMQ Artemis on WidlFly was working fine and some how this week since Monday I started having this exception quite frequently.
At first I went to the stack trace and looked into the code through code grep that throws this exception, and it seemed to me that the issue might be with the large size of the message. I stopped the process that inputs the message in the queues. and turned on my WildFly. The queue did not have data but I still got this exception.
I removed the ActiveMQ data directory and the tx-object-store from the data
directory. I started WildFly and it had the same issue, and I am still getting this exception. There was one site saying that may be the ports are wrongly defined, but I have been using this config for months and I don't seem to have changed anything in there nor have I upgraded WildFly 10 to another version so the libs are the same still.
Can any one tell me why is this error coming if there is no data flowing through ActiveMQ. I removed the data directory so there was nothing that is loaded for it to decode nor it is receiving anything to put on the queue to be decoded.
I turned up the log level in ActiveMQ Artemis and the exception looks like this, what is it getting in this log. I am not sending it any messages to process yet it is still processing something.
The issue seems to be from Netty, but I can't seem to point where my config would be wrong esp when I dont think they were changed anywhere.
2016-12-15 15:58:24,707 TRACE [org.apache.activemq.artemis.core.client] (Thread-18 (**activemq-netty-threads**-969652671)) handling packet PACKET(SessionCommitMessage)[type=43, channelID=12, packetObject=SessionCommitMessage]
2016-12-15 15:58:24,707 DEBUG [org.apache.activemq.artemis.core.client] (Thread-18 (activemq-netty-threads-969652671)) Invocation of interceptor org.apache.activemq.artemis.core.protocol.hornetq.HQPropertiesConversionInterceptor on PACKET(SessionCommitMessage)[type=43, channelID=12, packetObject=SessionCommitMessage] returned true
2016-12-15 15:58:24,707 TRACE [org.apache.activemq.artemis.core.server] (Thread-18 (activemq-netty-threads-969652671)) ServerSessionPacketHandler::handlePacket,PACKET(SessionCommitMessage)[type=43, channelID=12, packetObject=SessionCommitMessage]
2016-12-15 15:58:24,707 TRACE [org.apache.activemq.artemis.core.server] (Thread-18 (activemq-netty-threads-969652671)) Calling commit
2016-12-15 15:58:24,707 TRACE [org.apache.activemq.artemis.core.server] (Thread-18 (activemq-netty-threads-969652671)) TransactionImpl::commit::TransactionImpl [xid=null, id=31125, xid=null, state=ACTIVE, createTime=1481781503706(Thu Dec 15 15:58:23 AEST 2016), timeoutSeconds=300, nr operations = 0]@3cd1f6f3
2016-12-15 15:58:24,708 TRACE [org.apache.activemq.artemis.core.server] (Thread-18 (activemq-netty-threads-969652671)) ServerSessionPacketHandler::scheduling response::PACKET(NullResponseMessage)[type=21, channelID=0, packetObject=NullResponseMessage]
2016-12-15 15:58:24,708 DEBUG [org.apache.activemq.artemis.core.client] (Thread-18 (activemq-netty-threads-969652671)) Invocation of interceptor org.apache.activemq.artemis.core.protocol.hornetq.HQPropertiesConversionInterceptor on PACKET(NullResponseMessage)[type=21, channelID=0, packetObject=NullResponseMessage] returned true
2016-12-15 15:58:24,708 TRACE [org.apache.activemq.artemis.core.client] (Thread-18 (activemq-netty-threads-969652671)) Sending packet nonblocking PACKET(NullResponseMessage)[type=21, channelID=12, packetObject=NullResponseMessage] on channeID=12
2016-12-15 15:58:24,708 TRACE [org.apache.activemq.artemis.core.client] (Thread-18 (activemq-netty-threads-969652671)) Writing buffer for channelID=12
2016-12-15 15:58:24,864 TRACE [org.apache.activemq.artemis.core.server] (Thread-72 (activemq-netty-threads-969652671)) Connection created org.apache.activemq.artemis.core.remoting.impl.netty.NettyServerConnection@13a92998[local= /192.168.78.30:7045, remote=/10.10.16.11:47709]
2016-12-15 15:58:24,865 ERROR [org.apache.activemq.artemis.core.client] (Thread-72 (activemq-netty-threads-969652671)) AMQ214013: Failed to decode packet: java.lang.IllegalArgumentException: AMQ119032: Invalid type: 1
at org.apache.activemq.artemis.core.protocol.core.impl.PacketDecoder.decode(PacketDecoder.java:413)
at org.apache.activemq.artemis.core.protocol.ClientPacketDecoder.decode(ClientPacketDecoder.java:60)
at org.apache.activemq.artemis.core.protocol.ServerPacketDecoder.decode(ServerPacketDecoder.java:202)
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:324)
at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:605)
at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:68)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at io.netty.handler.codec.ByteToMessageDecoder.handlerRemoved(ByteToMessageDecoder.java:216)
at io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:527)
at io.netty.channel.DefaultChannelPipeline.callHandlerRemoved(DefaultChannelPipeline.java:521)
at io.netty.channel.DefaultChannelPipeline.remove0(DefaultChannelPipeline.java:351)
at io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:322)
at io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:299)
at org.apache.activemq.artemis.core.protocol.ProtocolHandler$ProtocolDecoder.decode(ProtocolHandler.java:174)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:349)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:244)
at org.apache.activemq.artemis.core.protocol.ProtocolHandler$ProtocolDecoder.channelRead(ProtocolHandler.java:117)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
My standalone.xml
:
<subsystem xmlns="urn:jboss:domain:messaging-activemq:1.0">
<server name="default">
<security enabled="false"/>
<management jmx-enabled="true"/>
<journal file-size="1024000"/>
<statistics enabled="true"/>
<shared-store-master/>
<security-setting name="#">
<role name="guest" delete-non-durable-queue="true" create-non-durable-queue="true" consume="true" send="true"/>
</security-setting>
<address-setting name="#" message-counter-history-day-limit="10" page-size-bytes="4194304" max-size-bytes="20971520" expiry-address="jms.queue.ExpiryQueue" dead-letter-address="jms.queue.DLQ"/>
<http-connector name="http-connector" endpoint="http-acceptor" socket-binding="http"/>
<http-connector name="http-connector-throughput" endpoint="http-acceptor-throughput" socket-binding="http">
<param name="batch-delay" value="50"/>
</http-connector>
<remote-connector name="netty" socket-binding="messaging"/>
<remote-connector name="netty-throughput" socket-binding="messaging-throughput">
<param name="batch-delay" value="50"/>
</remote-connector>
<in-vm-connector name="in-vm" server-id="0"/>
<http-acceptor name="http-acceptor" http-listener="default"/>
<http-acceptor name="http-acceptor-throughput" http-listener="default">
<param name="batch-delay" value="50"/>
<param name="direct-deliver" value="false"/>
</http-acceptor>
<remote-acceptor name="netty" socket-binding="messaging"/>
<remote-acceptor name="netty-throughput" socket-binding="messaging-throughput">
<param name="batch-delay" value="50"/>
<param name="direct-deliver" value="false"/>
</remote-acceptor>
<in-vm-acceptor name="in-vm" server-id="0"/>
<divert name="divert-to-multidm" forwarding-address="jms.queue.dbservice.multi_dm" address="jms.queue.dbservice.cap_tp_2"/>
<divert name="divert-to-multifo" forwarding-address="jma.queue.dbservice.multi_fo" address="jms.queue.dbservice.cl1test_odc_capfotp2"/>
<!--divert name="divert-to-cl1fo" forwarding-address="jma.queue.dbservice.cl1_fo" address="jms.queue.dbservice.cl1test_odc_capfotp2"/>
<divert name="divert-to-tmafocl1devfo" forwarding-address="jma.queue.dbservice.tmafocl1dev_fo" address="jms.queue.dbservice.cl1_test_odc_capfotp"/>
<divert name="divert-to-tmafocl1devdm" forwarding-address="jma.queue.dbservice.tmafocl1dev_dm" address="jms.queue.dbservice.cl1_test_odc_cap_tp"/-->
<jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>
<jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>
<jms-queue name="dbservice.cap_tp_2" entries="/queue/cap_tp_2"/>
<jms-queue name="dbservice.tp2_dm" entries="/queue/tp2_dm"/>
<jms-queue name="dbservice.tp2_fo" entries="/queue/tp2_fo"/>
<!--jms-queue name="dbservice.tmafocl1dev_fo" entries="/queue/tmafocl1dev_fo"/>
<jms-queue name="dbservice.tmafocl1dev_dm" entries="/queue/tmafocl1dev_dm"/-->
<connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>
<jms-queue name="dbservice.tmafocl1dev_dm" entries="/queue/tmafocl1dev_dm"/-->
<connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>
<connection-factory name="RemoteConnectionFactory" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector" />
<pooled-connection-factory name="hornetq-ra" transaction="xa" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" connectors="in-vm" min-large-message-size="10240"/>
</server>
</subsystem>
...
<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
<socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
<socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9993}"/>
<socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>
<socket-binding name="http" port="${jboss.http.port:8080}"/>
<socket-binding name="https" port="${jboss.https.port:9443}"/>
<socket-binding name="iiop" interface="unsecure" port="3528"/>
<socket-binding name="iiop-ssl" interface="unsecure" port="3529"/>
<socket-binding name="messaging" port="5445"/>
<socket-binding name="messaging-backup" port="5545"/>
<socket-binding name="messaging-throughput" port="5455"/>
<socket-binding name="txn-recovery-environment" port="4712"/>
<socket-binding name="txn-status-manager" port="4713"/>
<socket-binding name="node2-jms-broker" port="${node2.broker.port:5445}"/>
<socket-binding name="node2throughput-jms-broker" port="${node2throughput.broker.port:5455}"/>
<outbound-socket-binding name="mail-smtp">
<remote-destination host="localhost" port="25"/>
</outbound-socket-binding>
</socket-binding-group>