How do I group consumers in RabbitMQ?

Question

We are writing mail sync system, and we use RabbitMQ for that. Every producer pushes mails ids, then consumer gets ids and insert mails to db. In situation when we have 100 consumers (for example) and producers will generate ids too fast, every consumer will get ids and will use api to get mails, so then will be exception about limit of concurrent request to the api.

Сan we limit consumer for each producer ( for example, if max 3 consumer will be receive ids of one producer, then next 3 will receive from other one, and so on) ?

Answer 1

Сan we limit consumer for each producer ( for example, if max 3 consumer will be receive ids of one producer, then next 3 will receive from other one, and so on) ?

You could do this by using simple routing.

ProducerA sends messages with routing key routeA and consumer1, consumer2 and consumer3 are subscribed to exchange with routing key routeA.
ProducerB sends messages with routing key routeB and consumer4, consumer5 and consumer6 are subscribed to exchange with routing key routeB. .. and so on
You could also use topic exchange.

However, it seems to me that this may not be the solution to the problem of the exception about limit concurrent requests to the API. You didn't specify which API, so I can assume that this number is configurable and you can increase it, or simply the concurrent access is not allowed (which is hard to imagine since, you know, is not the 70s), in which case the whole idea of parallelism crumbles and falls...