5

I have a spring boot application, using spring-boot version 1.5.16.RELEASE, which adds the below dependencies to my app:

[INFO] |  +- com.rabbitmq:http-client:jar:1.0.0.RELEASE:compile
[INFO] |  +- org.springframework.amqp:spring-rabbit:jar:1.7.10.RELEASE:compile
[INFO] |  |  +- org.springframework.amqp:spring-amqp:jar:1.7.10.RELEASE:compile
[INFO] |  |  \- com.rabbitmq:amqp-client:jar:4.8.1:compile
[INFO] +- org.springframework.boot:spring-boot-starter-amqp:jar:1.5.16.RELEASE:compile

The application also uses rabbitmq server 3.6.6 as a message-broker.

I notice rarely that I receive the below error in my application:

org.springframework.amqp.AmqpIOException: java.io.IOException at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:71) at org.springframework.amqp.rabbit.connection.SimpleConnection.createChannel(SimpleConnection.java:67) at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createBareChannel(CachingConnectionFactory.java:1208) at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.access$200(CachingConnectionFactory.java:1197) at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.doCreateBareChannel(CachingConnectionFactory.java:562) at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createBareChannel(CachingConnectionFactory.java:545) at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getCachedChannelProxy(CachingConnectionFactory.java:515) at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getChannel(CachingConnectionFactory.java:497) at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.access$1300(CachingConnectionFactory.java:102) at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createChannel(CachingConnectionFactory.java:1213) at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1443) at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1419) at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:713) at org.springframework.amqp.rabbit.core.RabbitTemplate.send(RabbitTemplate.java:707) ... at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:899) at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:790) at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:105) at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:208) at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1381) at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:760) at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1324) at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:1294) at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1800(SimpleMessageListenerContainer.java:105) at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1550) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: null at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:124) at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:120) at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:142) at com.rabbitmq.client.impl.ChannelN.open(ChannelN.java:136) at com.rabbitmq.client.impl.ChannelManager.createChannel(ChannelManager.java:176) at com.rabbitmq.client.impl.AMQConnection.createChannel(AMQConnection.java:559) at org.springframework.amqp.rabbit.connection.SimpleConnection.createChannel(SimpleConnection.java:56) ... 43 common frames omitted Caused by: com.rabbitmq.client.ShutdownSignalException: connection error; protocol method: #method(reply-code=504, reply-text=CHANNEL_ERROR - second 'channel.open' seen, class-id=20, method-id=10) at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:66) at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:36) at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:443) at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:263) at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:136) ... 47 common frames omitted

The application consumes from a certain queue on ‍RabbitMQ, using 50 concurrent consumers. Also, the application runs on two nodes (servers) and consumes the same queue, with 50 consumers for each node.

The above error is rarely seen. To explain what I mean, I have about 30 instances of the above application (so 60 servers), each of which receives traffic of 100 requests per second (events consumed from RabbitMQ) and most of these produce outgoing events towards RabbitMQ, like the example in the stack trace above. The frequency that the error appears is 3-4 times a month (in all application instances and not in each one of them).

I searched through the internet for this error, thrown specifically from java amqp-client, but without luck. I also tried increasing the channel cache size (as suggested in documentation) so that all channels are cached on the client side, but the issue reappeared after this change. Could this be caused by the way I use RabbitMQ from my app, or is it something that is this an error that is not handled inside the RabbitMQ client?

saeed foroughi
  • 1,662
  • 1
  • 13
  • 25
sakis11
  • 51
  • 4
  • The problem is that you don't show how really you use RabbitMQ in your app. The point might be that you try to send withing listener and there is no transactional configuration, so your `RabbitTemplate` try to reopen the same channel. But that' all just speculation because we don't know your code. – Artem Bilan Feb 12 '19 at 16:54
  • @ArtemBilan Thank you for your comment. Indeed the listener thread is sending an outgoing message in this particular case. But as I explain above this is an error I get really rarely. Also, indeed I do not use transacted channels but the documentation says that the channels are cached despite they are transacted or not (https://docs.spring.io/spring-amqp/docs/1.7.10.RELEASE/reference/html/_reference.html#connections), so in my case I suppose that no physical close and reopen takes place, but instead rabbitTemplate will get the opened connection from the channels' cache. Am I missing something? – sakis11 Feb 14 '19 at 11:27
  • Your logic is correct, but somehow a channel is treated as closed and `RabbitTemplate` tries to reopen it. Consider to use a separate `ConnectionFactory` for the `RabbitTemplate`. Would be great to have some sample on GitHub to let us to play and reproduce if any. – Artem Bilan Feb 14 '19 at 14:28
  • @ArtemBilan Thank you for your suggestions. I already create a CachingConnectionFactory which I use in turn to create a LocalizedQueueConnectionFactory as a bean. This is what I use in my rabbit template. If I understand well, this what you suggest above. I have also upgraded all my rabbitmq servers to version 3.7.10 with the hope that the newer version will handle such errors. I will also increment my channelCache size since I have seen cases that my channel count per connection exceeds it. The error hasn't appeared for about 1 month now, without any obvious solution. – sakis11 Feb 28 '19 at 14:08

0 Answers0