3

I have set up a RabbitMQ for messaging, the integration works fine. I am sending messages to a queue and consuming from the queue. but how can I be sure about sending a message to the queue after performing a task?

For example, After I Add a new item into DataBase, I should publish a message to the queue.

DBContext.SaveChanges(item);

//what if an error occurred at this moment.
QueueManager.Publish(item)

What is the best practice in this situation? any suggestion will appreciate.

I read this solution Solution.

But I think there should be a better solution because there are two problems with this approach:

  1. in microservice architecture may some of the databases do not support atomic transaction
  2. it needs extra data manipulation and an extra background service for removing records from databases
Matt Qafouri
  • 1,449
  • 2
  • 12
  • 26
  • 1
    Save state on item in same transaction you save your item. Make publish if this state is in initial publish state and update this state after publish (which can also fail, but you can do more complex high load checks to ensure consistency, because it is separate process). – eocron Jun 28 '21 at 12:48
  • Does this answer your question? [Best way to ensure an event is eventually published to a message queuing sytem](https://stackoverflow.com/questions/30780979/best-way-to-ensure-an-event-is-eventually-published-to-a-message-queuing-sytem) – eocron Jun 28 '21 at 12:53

4 Answers4

3

how can I be sure about sending a message to the queue after performing a task?

Create the new table name Event, use the Event table which contains record of NonPublished events. Once done with the Database call, make an entry in the event table and mark that event as a NonPublished. And after publishing the event to queue, update that record as a Published event.

What happens if the application not able to publish the event(Queue system is down, the publisher is failed)?

Write a background job/cron job that runs periodically, which will check all Unpublished events and publish to queue and make the status as a Published. Also configure the retry mechanism, as mentioned by Tarun if the event published failed.

Please refer to the below link for more information about Event-driven microservices.

https://www.nginx.com/blog/event-driven-data-management-microservices/

Vaibs
  • 1,546
  • 9
  • 31
1

in microservice architecture may some of the databases do not support atomic transaction

I would think if you can be in position where publishing can fail, and the functionality is critical, then atomic transaction support cannot be optional. So, you try to publish, if it fails, rollback transaction. That means in the solution Best way to ensure an event is eventually published to a message queuing sytem, I would not even prefer to use async publish.

If the functionality is not critical then you can do all sort of optimization, including putting error messages in logs in case it fails to publish once in a while.

it needs extra data manipulation and an extra background service for removing records from databases

Normally, users prefer safe than sorry approach. So this should be okay when functionality is not critical.

Nimantha
  • 6,405
  • 6
  • 28
  • 69
0

You can use a retry pattern from Resiliency patterns of Microservices.

Retry – The source application can immediately retry to send the request to the service. If the specific fault is unusual or rare, the probability of success when repeating the request is very high.

Retry after a delay – The source application can retry to send the request to the cloud service after a period of time (that normally increases exponentially). This is a common practice when the failure event is due to reasons such as cloud service busy and so on.If the fault is caused by one of the more commonplace connectivity or busy failures, then the application has to wait for some time and try again.

Source : Retry Pattern

Tarun
  • 986
  • 6
  • 19
0

I don't know if I'm misinterpreting your question or the other people who answered your question have, but as I understand it, you want to publish a message to the queue if SaveChanges() has finished succesfully? Normally the method should return a boolean or the object you're saving so if that returns true/the object you can send the message.

Something like this:

if(DBContext.SaveChanges(item)) {
QueueManager.Publish(item)
} else {
// Error handling

}

Jonathazn
  • 11
  • 1
  • 2
  • What if we can successfully persist data into DB but in the publishing part we would face a problem, how to deal with that? In Kafka we do have transaction, but not in RabbitMQ – Matt Qafouri Jun 08 '23 at 07:11
  • Maybe RabbitMQ has error handling if the message couldn't be send for some reason? I haven't used RabbitMQ in a long time so I'm not familiar with their libraries. – Jonathazn Jun 09 '23 at 12:37