0

I checked "Message durability" part of RabbitMQ tutorial. But it has this note:

Marking messages as persistent doesn't fully guarantee that a message won't be lost. Although it tells RabbitMQ to save the message to disk, there is still a short time window when RabbitMQ has accepted a message and hasn't saved it yet. Also, RabbitMQ doesn't do fsync(2) for every message -- it may be just saved to cache and not really written to the disk

But what if I need really durable queue? Which best practices can I to use? Should I have "queue" in database and some "resender" by cron if, for example, message wasn't ACK'ed in 2 minutes? Are there some better solutions?

Also, what if my consumer crashed after it processed message and before it sent ACK?

UPD: my question was marked as "possible duplicate" of clustering question. I have no idea how clusters can help to solve these problems.

Guy Fawkes
  • 2,313
  • 2
  • 22
  • 39
  • http://stackoverflow.com/questions/29095990/rabbitmq-clustering – jpaljasma Oct 28 '15 at 16:01
  • What if whole cluster crashed? – Guy Fawkes Oct 28 '15 at 16:25
  • The probability is very low ... yet, if you are _really_ worried and don't need ACK at the application level, you could have your own queue for situations like that. Depending on application - perhaps tiny Redis instance installed on local web server + python / php via cron / node.js app to push send queue after RabbitMQ becomes available again? – jpaljasma Oct 28 '15 at 16:34

2 Answers2

2

Please read here about Guaranteed Delivery with Tx

The publisher would use something like:

ch.txSelect();
ch.basicPublish("", QUEUE_NAME,MessageProperties.PERSISTENT_BASIC,"nop".getBytes());
ch.txCommit();

Note: This can kill your performance application!

EDIT

Read also "Publisher Confirms" https://www.rabbitmq.com/confirms.html as suggesgted by @old_sound

Gabriele Santomaggio
  • 21,656
  • 4
  • 52
  • 52
  • > The operative word here is should, since the message can still be lost if broker goes down before it's had a chance to write the message to disk. – Guy Fawkes Oct 28 '15 at 16:24
  • Yes, and you can handle it from your client, since the commit fails – Gabriele Santomaggio Oct 28 '15 at 16:34
  • 1
    you will never have 100% reliable communication with any system. end of story. plan for failure and build a system that handles the most common and expected faults. rabbitmq will go down at some point. your system needs to know how to deal with that. most often, this is handled with in-memory or on-disk cache of messages on the publisher side, waiting for confirmation of send to remove from local cache – Derick Bailey Oct 28 '15 at 18:02
  • In try..catch I need some code to "re-send" message or broker will do it after restart/start after crash? – Guy Fawkes Oct 29 '15 at 07:36
  • It is called `HA client`, you have to handle your message, then re-try when the server is up. An `HA client` example is this: https://github.com/jhalterman/lyra – Gabriele Santomaggio Oct 29 '15 at 07:55
  • Are there some clients for other languages (python/php/ruby)? – Guy Fawkes Oct 29 '15 at 15:03
0

Take a look at Publisher Confirms that are made to address the problem of knowing if RabbitMQ has persisted/replicated your message or not: https://www.rabbitmq.com/confirms.html

old_sound
  • 2,243
  • 1
  • 13
  • 16