I try to connect to a RabbitMQ with spring boot. The connection should always restart/retry the connection. I have a problem reconnecting after a fatal exception. The application can never loose a connection and not indefinitely retry to get a connection.
@Bean
public IntegrationFlow flow() {
final SimpleMessageListenerContainer listenerContainer = new SimpleMessageListenerContainer(getConnectionFactory());
listenerContainer.setQueues(getQueue());
listenerContainer.setDeclarationRetries(Integer.MAX_VALUE);
final AmqpInboundChannelAdapterSpec adapter = (AmqpInboundChannelAdapterSpec) Amqp.inboundAdapter(listenerContainer);
return IntegrationFlows
.from(adapter)
.filter(filter)
.transform(transformer)
.handle(processor)
.get();
}
I may get a fatal Exception which I catch in a ApplicationListener<ListenerContainerConsumerFailedEvent>. This exception is not always the same, the last time it was a 'PossibleAuthenticationFailureException'
If the exception is fatal I stop and then start the container. I realize this may be the wrong place, as it looks like the container is stopped after calling of the event.
@Override
public void onApplicationEvent(ListenerContainerConsumerFailedEvent event) {
if (event.isFatal()) {
SimpleMessageListenerContainer simpleMessageListenerContainer = (SimpleMessageListenerContainer) event.getSource()
simpleMessageListenerContainer.stop();
simpleMessageListenerContainer.start();
LOG.info("started container");
}
}
resulting in the following output and a no connection to the rabbit. (After the following output nothing follows, it just does nothing is not connected)
[rContainer#0-34] startConnectionOnFatalConnectionListener : started container.
[rContainer#0-34] o.s.a.r.l.SimpleMessageListenerContainer : Stopping container from aborted consumer
Versions:
- RabbitMQ 3.6.8, Erlang 19.2
- Spring Integration (spring-integration-amqp): 4.3.9
I was able to get more log entries, the following lines get logged before the ListenerContainerConsumerFailedEvent is called:
07:05.843 ERROR o.s.a.r.c.CachingConnectionFactory : Channel shutdown: connection error; protocol method: #method<connection.close>(reply-code=541, reply-text=INTERNAL_ERROR, class-id=0, method-id=0)
07:05.843 ERROR o.s.a.r.c.CachingConnectionFactory : Channel shutdown: connection error; protocol method: #method<connection.close>(reply-code=541, reply-text=INTERNAL_ERROR, class-id=0, method-id=0)
07:15.087 ERROR o.s.a.r.l.SimpleMessageListenerContainer : Consumer received fatal exception on startup
org.springframework.amqp.rabbit.listener.exception.FatalListenerStartupException: Authentication failure
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:476) ~[spring-rabbit-1.6.1.RELEASE.jar!/:na]
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1280) ~[spring-rabbit-1.6.1.RELEASE.jar!/:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_92]
Caused by: org.springframework.amqp.AmqpAuthenticationException: com.rabbitmq.client.PossibleAuthenticationFailureException: Possibly caused by authentication failure
at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:65) ~[spring-rabbit-1.6.1.RELEASE.jar!/:na]
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:309) ~[spring-rabbit-1.6.1.RELEASE.jar!/:na]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:547) ~[spring-rabbit-1.6.1.RELEASE.jar!/:na]
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils$1.createConnection(ConnectionFactoryUtils.java:90) ~[spring-rabbit-1.6.1.RELEASE.jar!/:na]
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.doGetTransactionalResourceHolder(ConnectionFactoryUtils.java:140) ~[spring-rabbit-1.6.1.RELEASE.jar!/:na]
at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.getTransactionalResourceHolder(ConnectionFactoryUtils.java:76) ~[spring-rabbit-1.6.1.RELEASE.jar!/:na]
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:472) ~[spring-rabbit-1.6.1.RELEASE.jar!/:na]
... 2 common frames omitted
Caused by: com.rabbitmq.client.PossibleAuthenticationFailureException: Possibly caused by authentication failure
at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:342) ~[amqp-client-3.6.3.jar!/:na]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:813) ~[amqp-client-3.6.3.jar!/:na]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:767) ~[amqp-client-3.6.3.jar!/:na]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:887) ~[amqp-client-3.6.3.jar!/:na]
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:300) ~[spring-rabbit-1.6.1.RELEASE.jar!/:na]
... 7 common frames omitted
Caused by: com.rabbitmq.client.ShutdownSignalException: connection error
at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67) ~[amqp-client-3.6.3.jar!/:na]
at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:37) ~[amqp-client-3.6.3.jar!/:na]
at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:367) ~[amqp-client-3.6.3.jar!/:na]
at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:234) ~[amqp-client-3.6.3.jar!/:na]
at com.rabbitmq.client.impl.AMQChannel.rpc(AMQChannel.java:212) ~[amqp-client-3.6.3.jar!/:na]
at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:327) ~[amqp-client-3.6.3.jar!/:na]
... 11 common frames omitted
Caused by: java.io.EOFException: null
at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290) ~[na:1.8.0_92]
at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95) ~[amqp-client-3.6.3.jar!/:na]
at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:139) ~[amqp-client-3.6.3.jar!/:na]
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:542) ~[amqp-client-3.6.3.jar!/:na]
... 1 common frames omitted
Workaround: Implement a health check. The health check checks if the connection to rabbitMQ exists. The application get's restarted if the health check doesn't succeed. (Implemented as a cron, which calls a bash scripts. The bash scripts stops and start the container).
