Our production environment has an ActiveMQ server and 4 machines connecting to it, both as producers and consumers. Every 30 minutes, my ActiveMQ server logs display four lines(Corresponding to each of the four IPs connecting to the server)-
2018-11-18 11:32:19,081 | WARN | Transport Connection to: tcp://{IP1_here}:52614 failed: java.io.IOException: Frame size of 1140 MB larger than max allowed 100 MB | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///{IP1_here}:52614@61616<br>
2018-11-18 11:32:19,081 | WARN | Transport Connection to: tcp://{IP2_here}:52614 failed: java.io.IOException: Frame size of 1140 MB larger than max allowed 100 MB | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///{IP2_here}:52614@61616<br>
2018-11-18 11:32:19,081 | WARN | Transport Connection to: tcp://{IP3_here}:52614 failed: java.io.IOException: Frame size of 1140 MB larger than max allowed 100 MB | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///{IP3_here}:52614@61616<br>
2018-11-18 11:32:19,081 | WARN | Transport Connection to: tcp://{IP4_here}:52614 failed: java.io.IOException: Frame size of 1140 MB larger than max allowed 100 MB | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///{IP4_here}:52614@61616<br>
Please also note that the frequency of the above logs is 30 minutes. We are not expecting our messages to get so big and all four machines send/receive different kinds of messages from different queues.
Another thing worth noticing is that the frame size 1140 MB is constant across all the logs.
Also, the server seems to be running fine for now and does not crash. However, we have had a history of server crashing due to Out of memory issue. Hence, this issue is being analyzed carefully to eliminate it as a cause of any future OOM issue.
Why I suspect that the issue is at the server end and not the clients connecting to the AMQ server?
All four IPs run 6-8 different java programs connecting to the ActiveMQ server, which means I have 20-30 different java applications connecting to the server.
However, There are very consistently number of error messages every 30 minutes is only 4.
There are over 100000 messages being sent and received in a day. Our application logs do not show any error or warning corresponding to this. Thus, it is becoming very difficult to identify the cause of this warning or the message which caused this warning. We are using ActiveMQ-5.10.0.
