-1

I am trying to apply some flow control in my application which based on RabbitMQ.

A very narrow brief over my system:

  • There are some blue-workers that scan and input directory and publish messages to an exchange.
  • There are other red-workers that consume from this exchange (based on routing keys) and do whatever they do with the data and than delete it.

The data that is "stored" in the exchange is quite big and the time that takes to a worker to go over it is noticeable. After some time I am getting a memory warning from RabbitMQ which states that the memory usage is too high and all publishing operations are stopped.

I tried to increase the amount of memory Rabbitmq is using but it just postpone the problem in a few hours (of runtime). I also made the queues disk-based instead of ram-based but my disk got full instead.

Since my input is not that big I can live with a "large" input queue from which the blue-workers read their input. So I thought to try and set some "max length" on the link between the blue-workers and the exchange. I believe that I won't loose anything here since the real bottleneck of my system is the red-workers (by the way I declared the link between the red-workers and the exchange with a prefech_count=2).

After saying all that... I didn't manage to apply such a max-length :( I am using Pika to declare my queues and working with channels. I read this (https://www.rabbitmq.com/maxlength.html) but didn't manage to implement it in my code and I would like to see an example that uses this max-size flag.

Andy Thomas
  • 1,219
  • 3
  • 19
  • 33

1 Answers1

0

Exchanges do not store messages: queues do.

If you set up the max-length of a queue, setting the corresponding arguments in the queue_declare() call, the messages that won't fit it, will be trashed.

However the publisher will be able to publish the messages to the exchange without noticing it.

The producer can get notice about them failing using dead-lettering.

If you want some help with the code, include the code in the question.

Sigi
  • 4,826
  • 1
  • 19
  • 23
  • Oh I see. I now managed to do it by using the 'x-max-length' argument. However I can't loose messages so I tried to add the `{'x-overflow': 'reject-publish'}` argument as well but the return value of `channel.basic_publish` is always `True` even though I can see that the messages don't continue the full pipeline. Do you have any idea why? – Andy Thomas Feb 06 '19 at 12:41
  • Have you enabled the [publisher confirms](https://www.rabbitmq.com/confirms.html#publisher-confirms)? from the documentation about [overflow](https://www.rabbitmq.com/maxlength.html#overflow-behaviour), that config only indicated to rabbitmq which messages to throw away, the new ones instead of the old ones. If you get the basic.nack, the publisher would be notified that the message hasn't reached its intended destination (the queue). I'd assume based on your description that you have implemented some kind of wait on the publisher side to try it out again after a delay? – Olivier Feb 06 '19 at 14:14
  • @AndyThomas because messages always succeed to be published if the exchange is reachable. If your exchange has no queues bound to it messages get lost. If the queue is full, messages get lost. In case you want to never loose messages you need to use dead-lettering, or some sort of feedback from consumer to producer, to let the producer slow-down/wait. But this is too abstract - add a small working example to let us help you. – Sigi Feb 06 '19 at 15:04
  • @Sigismondo Thanks! I think I will indeed implement some feedback mechanism like you said. – Andy Thomas Feb 10 '19 at 12:19