2

I'm looking for a way to buffer messages received by the exchange as long as there is at least one queue bind to that exchange.

Is it supported by RabbitMQ?

Maybe there are some workarounds (I didn't find any).

EDIT

My use case:

  1. I've got one data producer (which reads real-time data from an external system)
  2. I've got one fanout exchange which receives data from the producer
  3. On system startup, there might be no consumer, but after a few moments, there should be at least one which creates his own queue and binds it to the exchange from 2.

The problem is this short time between step 2. and 3. where there are no queues bound to the exchange created in step 1.

Of course, it's an edge case and after system initialization queues and exchanges are bound and everything works as expected.

Why queues and bindings has to be created by consumers (not by the producer)? Because I need a flexible setup where I can add consumers without any changes in other components code (e.g. producer).

EDIT 2

I'm processing the output from another system which stores both real-time and historical data. There are the cases where I want to read historical data first (on initialization) and then continue to handle real-time data.

I may mislead you by saying that there are multiple consumers. In the case where I need a buffer on exchange there is only one consumer (which writes everything to time series DB as it appears in queue).

jmarceli
  • 19,102
  • 6
  • 69
  • 67

2 Answers2

2

The RabbitMQ team monitors this mailing list and only sometimes answers questions on StackOverflow.

Why queues and bindings has to be created by consumers (not by the producer)?

Queues and bindings can be created by producers or consumers or both. The requirement is that the exact same arguments are used when creating them if a client application tries to "re-create" a queue or binding. If different arguments are used, a channel-level error will happen.

As you have found, if a producer publishes to an exchange that can't route messages, they will be lost. Olivier's suggestion to use an alternate exchange is a good one, but I recommend you have your producers create queues and bindings as well.

Luke Bakken
  • 8,993
  • 2
  • 20
  • 33
  • Thanks for your response, but creating queues by producer doesn't make any sense in my case. Producers don't know the names of queues used by consumers (queues are bind dynamically to required exchanges when a consumer is created). The idea is creating a flexible system where I can bind any consumer I want to any of the exchanges populated by the producers. Alternate exchange solution seems to be the best choice. – jmarceli Feb 24 '18 at 22:15
1

If you mean to avoid throwing away messages because there is no destination configured for it, yes. You should look at alternate exchange. This assume that before (or when) you start (or when), the alternate exchange is created (would typically go for fanout) and a queue is binded to it (let's call it notroutedq). So the messages are not lost, they will be stored in notroutedq.

From there you can possibly setup a mechanism that would reprocess messages in that queue - reinjecting them into the main exchange most likely - once a given time has passed or when a binding has been added to your main exchange.

-- EDIT --

Thanks for the updated info. Could you indicate how long typically you'd expect the past messages to be useful to the consumers?

In your description, you mention real-time data and possibly multiple consumers coming and going. Based on that, I'm not sure how much of the data kept in the notroutedq would be of value, and with which frequency you'd expect to resend them to the consumers. The cases I had with alternate exchange where mostly focused on identifying missing bindings, so that one could easily correct the bindings and reprocess the messages without loss. If the number of consumers varies through time and the data content is real-time, I'd wonder a bit about the benefit of keeping the data.

Olivier
  • 2,571
  • 1
  • 17
  • 25
  • Thanks for your response I was hoping that there is some automatic buffer which will add all message to queue automatically after binding it to the exchange sth. like: [exchange]->[buffer]-(delayed binding)->[queue]. Your solution requires handling two queues by consumer firstly AE queue and then regular one, but maybe there is no other way :( – jmarceli Feb 23 '18 at 13:57
  • What you describe is I guess linked to a setup where the queue and binding is created by the consumer, which would explain the need for the buffering? I can think of at least 2 approaches to handle it, one mostly automatically in RabbitMQ, the other requiring some external system. If you confirm my assumption (or clarify the details of your use case) we (the community) might come up with simple ways to resolve it. – Olivier Feb 23 '18 at 14:11
  • I've added more info about my use case. Based on your response I think that you understand it correctly. – jmarceli Feb 23 '18 at 19:14
  • @jmarceli Added some more request for info – Olivier Mar 02 '18 at 06:41