Pattern for sending messages to topics when entity state changes

Question

We are using Azure Service Bus to notify subscribers when a certain entity in our application has changed to a certain state. Right now we're doing this right after we call dbContext.SaveChangesAsync():

1. dbContext.SaveChangesAsync()

2. topicClient.SendAsync(someMessage)

The problem I'm having is this: Say the dbContext.SaveChangesAsync() goes through fine, but for whatever reason the call to topicClient.SendAsync() throws an exception. Now the subscribers to this topic won't be aware of the entity's change in state.

I tried using TransactionScope, but that doesn't work because, as I gathered, Azure doesn't use the DTC.

(I could switch the order of the above 2 steps, but then if the message sends fine and the save fails, the message contains bogus data.)

Does anyone have any suggestions on how to handle this issue? It seems like one that should be common, but I can't find anything online. If someone could point me in the right direction, I'd appreciate it.

Thank you in advance.

Answer 1

As you've outlined, the order of operations won't matter because there's no overlapping transaction between the two services, database and messaging service. If the datastore you're using supports transactions (e.g. Azure SQL server) you could get away without using two-phase commit and look into implementing the Outbox pattern.

NServiceBus provides the pattern as a feature. You can download the outbox sample that shows how to use it with RabbitMQ. The transport can be replaced with Azure Service Bus transport to fulfil your requirement.

Disclosure: I contribute to NServiceBus.

Answer 2

You can implement pattern called poison message queue or dead letter queue. There are other known synonyms such as retry queue.

The idea is to try an operation and if it does not succeed, put some metainformation into a queue and re-try that operation later. In your case, after dbContext.SaveChangesAsync() is called, you can put all necessary information into a durable queue and have a handler which will process that queue and in some kind of ProcessMessage() you can handle calls of topicClient.SendAsync(someMessage).
For example, if call to service bus does not succeed, you can return queue item back into the queue for later processing.

Of course you can dedicate mentioned queue only for failed calls of topicClient.SendAsync(someMessage), which can significantly reduce its size. Order of operation is insignificant as you can put any metainformation, which makes it possible to make call topicClient.SendAsync(someMessage) first and then try to update database.

Answer 3

Not a direct answer to the problem you're facing but I will share how we've solved a similar problem in one of our applications where we had to write data to separate tables in Azure Table Storage. Because we were writing data to separate tables, we could not use Entity Batch Transaction functionality available in Azure Table Storage.

The way we have solved this problem is by implementing something similar to eventual consistency pattern.

What we do is instead of performing saving data directly in the tables (which would mean making multiple network requests any of which could fail), we send the data that needs to be saved to a queue (we used storage queue). If we're able to save the data in the queue, that means the data will eventually be available.

Then we wrote an Azure Queue Triggered Function. In that function we're saving data in the tables where we would need to save. Once all the operations are successful, the message gets automatically deleted by Function runtime. In case any of the operation fails, the message will be sent to the queue again and will be dequeued again.

Now one important thing to understand is that these save methods have to be idempotent. Let's say we're writing into 3 tables and write operation for 1st table succeeds but the write operation for 2nd table fails. Next time the Function is invoked, it will try to write to 1st table again and the code should be able to handle that gracefully.