How to save message into database and send response into topic eventually consistent?

Question

I have the following rabbitMq consumer:

Consumer consumer = new DefaultConsumer(channel) {
    @Override
     public void handleDelivery(String consumerTag, Envelope envelope, MQP.BasicProperties properties, byte[] body) throws IOException {
            String message = new String(body, "UTF-8");
            sendNotificationIntoTopic(message);
            saveIntoDatabase(message);
     }
};

Following situation can occur:

Message was send into topic successfully
Connection to database was lost so database insert was failed.

As a result we have data inconsistency.

Expected result either both action were successfully executed or both were not executed at all.

Any solutions how can I achieve it?

P.S.

Currently I have following idea(please comment upon)

We can suppose that broker doesn't lose any messages.

We have to be subscribed on topic we want to send.

Save entry into database and set field status with value 'pending'
Attempt to send data to topic. If send was successfull - update field status with value 'success'
We have to have a sheduled job which have to check rows with pending status. At the moment 2 cases are possible:
3.1 Notification wasn't send at all
3.2 Notification was send but save into database was failed(probability is very low but it is possible)

So we have to distinquish that 2 cases somehow: we may store messages from topic in the collection and job can check if the message was accepted or not. So if job found a message which corresponds the database row we have to update status to "success". Otherwise we have to remove entry from database.

I think my idea has some weaknesses(for example if we have multinode application we have to store messages in hazelcast(or analogs) but it is additional point of hypothetical failure)

@user7294900 we have limited count of retries. If the broker is down we can exhaust all attempts and we again have data inconsistency — gstackoverflow, Jul 03 '19 at 10:45
@user7294900 I don't know) But I didn't encounter systems with 1 billions retry attempts — gstackoverflow, Jul 03 '19 at 10:48
The solution is to use a messaging system that supports JMS and XA transactions, and to use an XA transaction manager. Or to have business logic that is tolerant to inconsistencies. — JB Nizet, Jul 03 '19 at 10:54
@JB Nizet when someone hear smth about XA transaction he usually becomes nervous) — gstackoverflow, Jul 03 '19 at 11:08
@JB Nizet it sounds interesting but I can't imagine how to do it — gstackoverflow, Jul 03 '19 at 11:16
@JBNizet I don|t think XA is nessesary here. There are various compensation options. XA can be justified if you need a really high degree of consistency which I don't think is a so usual case. — Alexander Petrov, Jul 12 '19 at 10:21

Alexander Petrov · Accepted Answer · 2019-07-12T10:14:07.010

1

Here is an example of Try Cancel Confirm pattern https://servicecomb.apache.org/docs/distributed_saga_3/ that should be capable of dealing with your problem. You should tolerate some chance of double submission of the data via the queue. Here is an example:

Define abstraction Operation and Assign ID to the operation plus a timestamp.
Write status Pending to the database (you can do this in the same step as 1)
Write a listener that polls the database for all operations with status pending and older than "timeout"
For each pending operation send the data via the queue with the assigned ID.
The recipient side should be aware of the ID and if the ID has been processed nothing should happen.

6A. If you need to be 100% that the operation has completed you need a second queue where the recipient side will post a message ID - DONE. If such consistency is not necessary skip this step. Alternatively it can post ID -Failed reason for failure.

6B. The submitting side either waits for a message from 6A of completes the operation by writing status DONE to the database.

Once a sertine timeout has passed or certain retry limit has passed. You write status to operation FAIL.
You can potentialy send a message to the recipient side opertaion with ID rollback.

Notice that all this steps do not involve a technical transactions. You can do this with a non transactional database.

What I have written is a variation of the Try Cancel Confirm Pattern where each recipient of message should be aware of how to manage its own data.

edited Jul 12 '19 at 10:14

answered Jul 12 '19 at 10:08

Alexander Petrov

9,204
31
70

@gstackoverflow it is more detailed. You don't consider the possibility that you may post to a queue and still the message not being delivered to the recipient, or being delivered but not processes-ed due to application error or validation rule or something else... Posting successfully on a que does no necessarily means success. – Alexander Petrov Jul 12 '19 at 11:46
@gstackoverflow also you are not taking into account diverse compansating scenarious that may occure in case of error. – Alexander Petrov Jul 12 '19 at 11:47
job is self compensated. It will restart in case of failure – gstackoverflow Jul 12 '19 at 11:58
@gstackoverflow it is not. Successfull post on a queue does not mean there is NO error. You can end up in a situation where the message is consumed and yet the operation on the recipient side has not completed in a good way. How are you going to capture this ? Following your 3 steps you will write to the database Success after successfull message post. Your algorithm does no guarantee consistent state accross recipient and sender. – Alexander Petrov Jul 12 '19 at 12:39
there might be a topic. I might don't want to check that all topic subscribers were able to handle message – gstackoverflow Jul 12 '19 at 12:47
@gstackoverflow then it is fine. The solution complexity depends on how much consistancy and guarantees you want to achieve. – Alexander Petrov Jul 12 '19 at 13:11

score 0 · Answer 2 · answered Jul 03 '19 at 13:01

In the listener save database row with field staus='pending'
Another job(separated thread) will obtain all pending rows from DB and following for each row:
2.1 send data to topic
2.2 save into database

If we failured on the step 1 - everything is ok - data in consistent state because job won't know anything about that data

if we failured on the step 2.1 - no problem, next job invocation will attempt to handle it

if we failured on the step 2.2 - If we failured here - it means that next job invocation will handle the same data again. From the first glance you can think that it is a problem. But your consumer has to be idempotent - it means that it has to understand that message was already processed and skip the processing. This requirement is a consequence that all message brokers have guarantees that message will be delivered AT LEAST ONCE. So our consumers have to be ready for duplicated messages anyway. No problem again.

Krishna · Answer 3 · 2019-07-12T10:59:06.317

If there is enough time to modify the design, it is recommended to use JTA like APIs to manage 2phase commit. Even weblogic and WebSphere support XA resource for 2 phase commit.

If timeline is less, it is suggested perform as below to reduce the failure gap.

Send data topic (no commit) (incase topic is down, retry to be performed with an interval)
Write data into DB
Commit DB
Commit Topic

Here failure will happen only when step 4 fails. It will result in same message send again. So receiving system will receive duplicate message. Each message has unique messageID and CorrelationID in JMS2.0 structure. So finding duplicate is bit straight forward (but this is to be handled at receiving system)

Both case will work for clustered environment as well.

Strict to your case, thought below steps might help to overcome your issue

Subscribe a listener listener-1 to your topic.

Process-1

Add DB entry with status 'to be sent' for message msg-1
Send message msg-1 to topic. Retry sending incase of any topic failure If step 2 failed after certain retry, process-1 has to resend the msg-1 before sending any new messages OR step-1 to be rolled back

Listener-1

Using subscribed listener, read reference(meesageID/correlationID) from Topic, and update DB status to SENT, and read/remove message from topic. Incase reference-read success and DB update failed, topic still have message. So next read will update DB. Incase DB update success and message removal failed. Listener will read again and tries to update message which is already done. So can be ignored after validation.

Incase listener itself down, topic will have messages until listener reading the messages. Until then SENT messages will be in status 'to be sent'.

It won't work if we can't commit topic as you mentioned. It can't be considered as a solution — gstackoverflow, Jul 10 '19 at 13:37

Arpan Kanthal · Answer 4 · 2019-07-11T23:37:21.500

Here's the pseudocode for how i'd do it: (Assuming the dao layer has transactional capability and your messaging layer doesnt)

    //Start a transaction
    try {
                String message = new String(body, "UTF-8");
               // Ordering is important here as I'm assuming the database has commit and rollback capabilities, but the messaging system doesnt. 
                saveIntoDatabase(message);
                sendNotificationIntoTopic(message);

    } catch (MessageDeliveryException) {
        // rollback the transaction
        // Throw a domain specific exception
    }
   //commit the transaction

Scenarios:
1. If the database fails, the message wont be sent as the exception will break the code flow .
2. If the database call succeeds and the messaging system fails to deliver, catch the exception and rollback the database changes

All the actions necessary for logging and replaying the failures can be outside this method

@gstackoverflow in that case, the transaction isnt commited yet, so its safe — Arpan Kanthal, Jul 17 '19 at 00:26

How to save message into database and send response into topic eventually consistent?

P.S.

4 Answers4