11

I've got a project where we are going to have hundreds (potentially thousands) of queues in rabbit and each of these queues will need to be consumed by a pool of consumers.

In rabbit (using spring-amqp), you have the rabbitlistener annotation which allows me to statically assign the queues this particular consumer(s) will handle.

My question is - with rabbit and spring, is there a clean way for me to grab a section of queues (lets say queues that start with a-c) and then also listen for any queues that are created while the consumer is running.

Example (at start):

  • ant-queue
  • apple-queue
  • cat-queue

While consumer is running:

  • Add bat-queue

Here is the (very simple) code I currently have:

    @Component
    public class MessageConsumer {

        public MessageConsumer() {
            // ideally grab a section of queues here, initialize a parameter and give to the rabbitlistener annotation
        }

        @RabbitListener(queues= {"ant-queue", "apple-queue", "cat-queue"})
        public void processQueues(String messageAsJson) {
            < how do I update the queues declared in rabbit listener above ? >
        }
    }

Edit:

I should add - I've gone through the spring amqp documentation I found online and I haven't found anything outside of statically (either hardcoded or via properties) declaring the queues

user10776719
  • 301
  • 4
  • 15

1 Answers1

16

  • Inject (@Autowired or otherwise) the RabbitListenerEndpointRegistry.

  • Get a reference to the listener container (use the id attribute on the annotation to give it a known id) (registry.getListenerContainer(id)).

  • Cast the container to an AbstractMessageListenerContainer and call addQueues() or addQueueNames().

Note that is more efficient to use a DirectMessageListenerContainer when adding queues dynamically; with a SimpleMessageListenerContainer the consumer(s) are stopped and restarted. With the direct container, each queue gets its own consumer(s).

See Choosing a container.

kosgeinsky
  • 508
  • 3
  • 21
Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • 1
    "With the direct container, each queue gets its own consumer(s)" - music to my ears. Thank you for the info! – user10776719 Jan 08 '19 at 17:50
  • Gary - my mistake. Is it possible to asynchronously determine queues that exist within a rabbitmq server based on a certain naming convention? In theory, we would need to poll every second to determine which queues exist and create a consumer for it. – user10776719 Jan 11 '19 at 21:15
  • 1
    Don't ask new questions in comments to old answers, it won't help other people when searching for answers; ask a new question instead. There is nothing in the AMQP protocol to do that; RabbitMQ does provide a REST API (and a [java binding for it](https://docs.spring.io/spring-amqp/reference/html/_reference.html#management-rest-api)), but I would suggest that polling (especially at that rate) would not be a good idea. Consider adding a consumer for a queue bound to the [event exchange](https://www.rabbitmq.com/event-exchange.html) instead; you can get `queue.created` events. – Gary Russell Jan 11 '19 at 21:27
  • Thanks Gary - apologies for the question in comment – user10776719 Jan 11 '19 at 21:33
  • @GaryRussell could you please share any example of this. It is exactly what we are looking for, however we want a hybrid approach where some queue names are known in advance and some new queues will be discovered at runtime. – user1549994 Jul 29 '20 at 19:25
  • It is better to ask a new question showing your config so I can understand how you are using the framework to give you the best answer. – Gary Russell Jul 29 '20 at 21:13
  • Thank you for your quick response @GaryRussell. I have created a new question here: https://stackoverflow.com/questions/63162812/rabbitmq-dynamic-addition-of-queues-to-a-listener-at-runtime – user1549994 Jul 29 '20 at 21:38
  • @GaryRussell registry.getListenerContainer(id) and adding a queue name to it takes a long time, and I'm adding queue names in the container from another rabbitlistener. Isn't there a faster approach or any other way to make it faster? – parsa Apr 25 '22 at 15:20
  • It's best to ask a new question, with much more information, rather than commenting on a 3 year old answer. That said, using a `DirectMessageListenerContainer` will probably be quite a bit faster; aside from that, I am not aware of any issues. – Gary Russell Apr 25 '22 at 15:24
  • Is there an example of an implementation of this reply? – Tobia Jun 08 '23 at 20:15
  • Which reply? Why don't you ask a new question with more clarity; I can then add an answer with an example if needed. The admins here don't like prolonged discussions in comments on old answers. – Gary Russell Jun 08 '23 at 20:23