1

How to guarantee message delivery in a non-transacted, lightweight environment?

For example:

  • Normal situation: Write to database, commit, send message to ZeroMQ|Redis|OtherMQ, consumer pulls the message to continue processing...
  • 0,05% situation: Write to database, commit, application dies!, no message sent, no consumer pull the message, incomplete processing.

How to not loose the message (avoid not send the message) in this situation?

Edit: The message must be delivery exactly once.

ethanxyz_0
  • 713
  • 12
  • 37
  • Mark message in the db as "sent" when producer sends it, fetch all "not sent" messages on start up, not? There is not other way unless you can store somewhere information about not sent messages. – cassandrad Oct 26 '17 at 08:10
  • @cassandrad the app can die after the message was sent to MQserver but before the app marks the message was sent in db. Other option: in the same of db transaction the message (in db table) is still waiting for consumer. Commit, send message, consumer pulls the message and the first thing the consumer must do is mark the message as received in db. But I think this approach is slow for high frequence messages. – ethanxyz_0 Oct 26 '17 at 08:26
  • "he app can die after the message was sent to MQserver but before the app marks the message was sent in db". No problem at all, the app will send the message again when started up, no? You didn't specify delivery option should be exactly once. – cassandrad Oct 26 '17 at 08:28
  • But if the message was sent again, the same message will be consumed twice or more. The message must be delivery exactly once. – ethanxyz_0 Oct 26 '17 at 08:51
  • Then you need to specify that in your question, because it is important. – cassandrad Oct 26 '17 at 08:54
  • @cassandrad I agree. question edited. – ethanxyz_0 Oct 26 '17 at 09:03
  • I don't understand "non-transacted" environment because you write to database and commit. The changes are not applied to database before you commit? Could you please clarify what you mean by non-transacted? – Ali Sağlam Oct 26 '17 at 12:31
  • Hi @AliSağlam, only the database is transacted. The MQ server not. Wich means, if I need to post a message in ZeroMQ **after** the database commit, the application have no guarantee thats my message will be sent because the if the app dies, the database is updated, but the message was not sent. In a Full-XA environment, the database commit and the message producer will be "performed" together. – ethanxyz_0 Oct 26 '17 at 23:25
  • How about having a ReplayServer process to add this n-times Quality of Service to your message bus ? If you are on a topic, simply let it subscribe to that same topic. If on a queue, forward to bridged queue and this server would then store messages safely on disk without having. Whenever you need a message again, ask for it to be replayed. – Axel Podehl Jan 04 '18 at 08:54
  • I like your idea @AxelPodehl but the increase of the latency may be an issue.. – ethanxyz_0 Jan 12 '18 at 22:00
  • The latency of the out-of-band persisted message might be bad, but the original message going to the intended subscriber shouldn't increase in latency if the broker's architecture is good. You could make a test with a non-transacted, non-persistent scenario: what's the difference in latency of the first subscriber when you add one more subscriber ? Once the copy is received it could be stored in parallel, even on another machine, even with batch processing for high performance, – Axel Podehl Jan 15 '18 at 08:29

2 Answers2

3

In this scenario you have 2 shared resources (database and queue) and you want them to be transacted together. You want your database to commit if the message sent to the queue. You want your database not to commit if it is not successfully sent and vice versa. This is simply global transaction mechanism like 2PC. However to implement a global transaction mechanism is not that easy and it is also very costly.

I suggest you to implement at least one strategy on producer side and idempotency on consumer side to provide consistency.

You should create a message table on producer side's database and persist messages to this table before sending to queue. Then with a scheduled thread (here there may have multiple threads to increase throughput but be careful if your messages needs to be consumed in the order they produced) or any other thing you can send them to queue and mark them as sent to ensure that the messages which are already sent will not be sent again. Even if you do that there might be some cases that your messages are sent more than once (e.g. you send a message to queue and your application crashed before marking the message as sent). But it is not a problem, because we already want to implement at least once strategy on producer side which means we want a message to be sent to queue at least once.

To prevent a consumer to consume same messages which are produced more than once on producer side you should implement idempotent consumers. Simply, you can save id of consumed messages to a database table on consumer side and before processing messages coming from the queue, you may check if it is already consumed. If it is already consumed you should ignore it and get the next message.

There are of course other options to provide consistency in microservices environment. You can find other solutions on this great blog - https://www.nginx.com/blog/event-driven-data-management-microservices/. The solution i explained above also exists in this blog. You can find it in Publishing Events Using Local Transactions section.

Ali Sağlam
  • 1,410
  • 1
  • 10
  • 15
0

Here maybe a simple approach.

Let’s assume you have the transaction:

  1. write data to DB
  2. send message over ZMQ
  3. write to DB that sending was OK

So assume your app crash while you are in step 2 or 3. If so, you don’t know if the last message did receive the customers queue and you have to resend after restart all messages without the last confirmation (step 3).

The problem is on the consumer side, because it’s possible, that they receives a message twice. To solve this problem, you can send with each message an transaction-ID which is always increasing. The consumer have to notice the transaction-ID of the last message. When the incoming message have a transaction-ID that is not higher than the transaction-ID of the last message the message can be ignored.

The question is now if you can modify the message structure and which transaction-ID you can use.

RomanE
  • 101
  • 5