Failed to declare anonymous queue after rabbitmq in HA mode restart

Question

Rabbitmq is deployed on k8s cluster with following HA configuration.

{"ha-mode":"exactly","ha-params":2}

Spring boot application uses spring cloud stream(3.2.2) and has consumer for which anonymous queue is declared

@Configuration
public class EventConsumer {
@Bean
public Consumer<Message<Event>> consumeEvent() {
    return message -> {
        log.info("Message received");
    };
 }
}

Yaml

spring:
  rabbitmq:
    host: rabbitmq.rabbitmq #internal k8s DNS
    username: user
    password: pass
  cloud:
    stream:
      rabbit:
        bindings:
          consumeEvent-in-0:
            consumer:
              exchange-type: fanout
      bindings:
        consumeEvent-in-0:
          destination: destination

When node of rabbitmq on which given queue is declared goes down, application starts to throw following errors:

Failed to declare queue: destination.anonymous.CK8YMdieSmC6UGkczGsVRw

and

Queue declaration failed; retries left=n

org.springframework.amqp.rabbit.listener.BlockingQueueConsumer$DeclarationException: Failed to declare queue(s):[destination.anonymous.CK8YMdieSmC6UGkczGsVRw]
    at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.attemptPassiveDeclarations(BlockingQueueConsumer.java:743)
    at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.passiveDeclarations(BlockingQueueConsumer.java:620)
    at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:607)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.initialize(SimpleMessageListenerContainer.java:1350)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1195)
    at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: null
    at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:129)
    at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:125)
    at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:147)
    at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:1012)
    at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:46)
    at jdk.internal.reflect.GeneratedMethodAccessor162.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.base/java.lang.reflect.Method.invoke(Unknown Source)
    at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:1157)
    at jdk.proxy2/jdk.proxy2.$Proxy262.queueDeclarePassive(Unknown Source)
    at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.attemptPassiveDeclarations(BlockingQueueConsumer.java:721)
    ... 5 common frames omitted
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'destination.anonymous.CK8YMdieSmC6UGkczGsVRw' in vhost '/', class-id=50, method-id=10)
    at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:66)
    at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:36)
    at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:502)
    at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:293)
    at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:141)
    ... 13 common frames omitted
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'destination.anonymous.CK8YMdieSmC6UGkczGsVRw' in vhost '/', class-id=50, method-id=10)
    at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:517)
    at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:341)
    at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:182)
    at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:114)
    at com.rabbitmq.client.impl.AMQConnection.readFrame(AMQConnection.java:739)
    at com.rabbitmq.client.impl.AMQConnection.access$300(AMQConnection.java:47)
    at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:666)
    ... 1 common frames omitted

What I found is that for anonymous(auto-delete) queue additional argument

x-queue-locator-master:client-local

is set and because of that if node where given queue is declared goes down, Spring should redeclare it on the next node it connects to. However in this case Spring is not able to declare queue again. Can anybody help what is missing on application or rabbitmq configuration side to make this working? Expected behaviour is that Spring can declare anonymous queue again on next node where it connects to in case when previous node where given queue was declare went down.

It should be `client-local` not `client-side`. Spring uses an `AnonymousQueue` which sets it to that; if it is really `client-side` then something else is doing that. If it is `client-local` it should work as described. If the value is correct an it is not working, I suggest you reach out to the RabbitMQ engineers on the rabbitmq-users Google group; they don't monitor Stack Overflow. You should also check policies on the broker to make sure the setting is not getting overridden there. — Gary Russell, Sep 27 '22 at 13:23
Thank you for the answer. It is client-local indeed, my mistake(corrected already) and it is set by `AnonymousQueue` as you mentioned. I'll write to rabbitmq-users with this question — kbit8, Sep 27 '22 at 14:05
If the queue resides on a different node to one to which we are connected, then Spring gets no notification of that node going down. If the node we are connected to goes down, we will definitely re-declare the queue on the new node. With that setting, it should not be possible to get one of these queues on a node that is different to the one we are connected to. Make sure that there is no policy overriding the queue argument - look at the queue with the UI when it is present to look at its arguments. — Gary Russell, Sep 27 '22 at 15:48
Thank you Gary for your effort, I found what was the issue and posted the answer. Everything works as you described. The issue lied in other exchange declaration exception(PRECONDITION_FAILED). Misleading for me was that failure in exchange declaration was not interrupted queue declaration during application startup, but interrupted queue declaration when application was running and rabbitmq node went down and Spring tried to redeclare it. — kbit8, Sep 28 '22 at 09:19
You can set `ignoreDeclarationExceptions` to true on the `RabbitAdmin` and it will continue, rather than throwing an exception. — Gary Russell, Sep 28 '22 at 14:12

score 0 · Answer 1 · answered Sep 28 '22 at 09:10

Issue was related to exchange declaration which was declared along with anonymous queue. In RabbitAdmin there is a code which looks like this:

this.rabbitTemplate.execute(channel -> {
            declareExchanges(channel, exchanges.toArray(new Exchange[exchanges.size()]));//exception was thrown
            declareQueues(channel, queues.toArray(new Queue[queues.size()]));
            declareBindings(channel, bindings.toArray(new Binding[bindings.size()]));
            return null;
        });

In case when rabbitmq node went down new connection is established and RabbitAdmin#initialize is triggered which above code is part of. During exchange declaration exception was thrown and declareQueues was not called. So in RabbitAdmin queue was not declared but in BlockingQueueConsumer passiveDeclarations() was called and then DeclarationException was thrown

You can set `ignoreDeclarationExceptions` to true on the `RabbitAdmin` and it will continue, rather than throwing an exception. — Gary Russell, Sep 28 '22 at 14:12
I saw this option in `RabbitAdmin`, however not sure how can I set this because now it is auto configured by Spring. — kbit8, Sep 29 '22 at 15:21
You can get a reference to the auto configured admin as a parameter in some other bean definition and set the property there. — Gary Russell, Sep 29 '22 at 16:34

Failed to declare anonymous queue after rabbitmq in HA mode restart

1 Answers1