1

In some exceptional situations I need somehow to tell consumer on receiving point that some messages shouldn’t be processed. Otherwise two systems will become out-of-sync (we deal with some outdates external systems, and if, for example, connection is dropped we have to discard all queued operations in scope of that connection).

Take a risk and resolve problem messages manually? Compensation actions (that could be tough to support in my case)? Anything else?

Tenek
  • 437
  • 1
  • 6
  • 15
  • RabbitMQ has message expiration, but it sounds like you have things that are more stateful in your queues. I might suggest a database queue would provide a bit more usefulness in this type of a scenario. – theMayer Aug 14 '18 at 23:35
  • 1
    Expanding "Take a risk and resolve problem manually", consider Saga state machine as an option. Assuming those consumers already received you message, You could raise an event when connection dropped and make a transition to WaitForCleanUpState in you state machine an then do the clean up job. – Mohammad Reza Sadreddini Aug 15 '18 at 04:27
  • @MohammadRezaSadreddini yes, a Saga with WaitForCleanUpState is one of the options. – Tenek Aug 16 '18 at 15:45

3 Answers3

3

There are a few ways:

  • You can set a time-to-live when sending a message: await endpoint.Send(myMessage, c => c.TimeToLive = TimeSpan.FromHours(1));, but this will apply to all messages that are sent (or published) like this. I would consider this, after looking at your requirements. This is technical, but it is a proper messaging pattern.

  • Make TTL and generation timestamp properties of your message itself and let the consumer decide if the message is still worth processing. This is more business and, probably, the most correct way.

  • Combine tech and business - keep the timestamp and TTL in message headers so they don't pollute your message contracts, and filter them out using a custom middleware. In this case, you need to be careful to log such drops so you won't be left wonder why messages disappear now and then.

  • Almost any unreliable integration can be monitored using sagas, with timeouts. For example, we use a saga to integrate with Twilio. Since we have no ability to open a webhook for them, we poll after some interval to check the message status. You can start a saga when you get a message and schedule a message to check if the processing is still waiting. As discussed in comments, you can either use the "human intervention required" way to fix the issue or let the saga decide to drop the message.

  • A similar way could be to use a lookup table, where you put the list of messages that aren't relevant for processing. Such a table would be similar to the list of sagas. It seems that this way would also require scheduling. Both here, and for the saga, I'd recommend using a separate receive endpoint (a queue) for the DropIt message, with only one consumer. It would prevent DropIt messages from getting stuck behind the integration messages that are waiting to be processed (and some should be already dropped)

  • Use RMQ management API to remove messages from the queue. This is the worst method, I won't recommend it.

Alexey Zimarev
  • 17,944
  • 2
  • 55
  • 83
  • Thanks! I’m considering option two with timestamps check in consumer. Option one is possible I guess, but I’m not sure if it allows me to capture expiration event in my saga? Expiration is covered, but what about if message needs to be canceled before expiring (in our case it may happen if remote system drops connection)? That probably will hardly happen. Should I consider WaitForCleanUpState state for manual cleanup suggested by Mohammad Reza Sadreddini? Or use some additional state lookup outside RMQ (which will be less efficient)? – Tenek Aug 15 '18 at 17:21
  • The question contains no details about the saga, maybe you can elaborate. – Alexey Zimarev Aug 15 '18 at 18:55
  • If the message needs to be expired before the expiration date, you need to figure out how to identify such message. Definitely, you need to know the message id. If you know that - you can build a small table or something that will contain ids of messages that need to be dropped. Then, in your consumer, you can query this table. Alterntively, you can create a middleware that will filter them out for you. The same can be done for TTL (middleware). – Alexey Zimarev Aug 15 '18 at 18:57
  • Sounds like we have two options: 1. Use saga to manage Message’s State property. Where WaitForCleanUp would happen if message gets canceled for some reason after Received and before Expired or Processed states. This would require human intervention. Should be OK for rare occasions? 2. Sharing lookup table to specify what messages needs to be dropped is a good idea too, but my consumer sits on the other system and table can be shared DB or can messages ids be sent to consumer over other high priority channel (not very familiar with RMQ, maybe it doesn’t make sense). – Tenek Aug 15 '18 at 19:25
  • 1
    You can try a saga but it might be handled simpler using a lookup table for messages that need to be dropped. But yes, a saga would work too. Concerning priorities - you just need separate receive endpoint in MT, with different queues, for these messages. Then, you will process cancellation messages before consuming the actual message that need to be dropped, which might still wait in another queue. – Alexey Zimarev Aug 16 '18 at 08:52
  • Alexey, two queues is a great idea to avoid shared DB for looking up. The first main queue may become bottleneck and messages may end up in the queue for a few seconds. But, the second queue will be used only for telling consumer if some queued messages need to be dropped (consuming system will maintain own lookup table for this). Since it will be very lightly (hardly) used, I assume RMQ will deliver message almost immediately? If you agree, can you please extend your answer for dropping non-expired messages and I will accept it? – Tenek Aug 16 '18 at 15:35
  • 1
    can you also mention a Saga with manual intervention state as an option too for completeness :)? – Tenek Aug 16 '18 at 15:51
  • @Tenek I tried, hopefully I grasped the gist of our design discussion :) – Alexey Zimarev Aug 16 '18 at 17:38
0

From what I understand, you're building a system that sends messages to 3rd party systems. In other words, systems you don't control. It has an API but compensating actions aren't always possible, because the API doesn't provide it or because actions are performed inside the 3rd party system that can't be compensated or rolled back?

If possible try to solve this via sagas. Make sure the saga executes the different steps (the sending of messages) in the right order. So that messages that cannot be compensated are sent last. This way message that can be compensated if they fail, will be compensated by the saga. The ones that cannot be compensated should be sent last, when you're as sure as possible that they don't have to be compensated. Because that last message is the last step in synchronizing all systems.

All in all this is one of the problems with distributed systems, keeping everything in sync. Compensating actions is the way to deal with this. If compensating actions aren't possible, you're in a very difficult situation. Try to see if the business can help by becoming more flexible and accepting that you need to compensate things, where they'll tell you it's not possible.

Dennis van der Stelt
  • 2,203
  • 16
  • 22
0

In some exceptional situations I need somehow to tell consumer on receiving point that some messages shouldn’t be processed.

Can't you revert this into:

Tell the consumer that an earlier message can be processed.

This way you can easily turn this in a state machine (like a saga) that acts on two messages. If the 2nd message never arrives then you can discard the 1st after a while or do something else.

The strategy here is to halt/wait until certain that no actions need to be reverted.

Ramon Smits
  • 2,482
  • 1
  • 18
  • 20