0

We have to create a responder application as a windows service which will take messages off the queue, acknowledge the sender that we received them and that they are validated by sending a response back within 5 seconds. After we get another acknowledgement from the sender that we are okay to process the message, we process the message and send the result of that request back to them within another (5 sec). If at any point in that cycle either they or we do not get the message off the queue and respond within the time allotted the message will expire and we will need to re-try the message.

What we are having trouble with determining when messages will expire and need to be re-sent. I've been reading about putting them on the dead-letter queue, but I'm unsure that the purpose of it matches what we need. The dead-letter queue is not for items which just timeout, but failed in transmission. I've also read about using the Report option on the MQMessage itself to generate a report and possibly move it to a different queue which can be monitored. We'd then need to re-try the items that are on the new queue.

Are those my only two options if I need to monitor every incoming and outgoing message to ensure that we get a response from the other MQ server we are communicating with?

ProgrammingLlama
  • 36,677
  • 7
  • 67
  • 86
Jimenemex
  • 3,104
  • 3
  • 24
  • 56
  • You should build the Response/Ack logic in code rather than hoping to configure MQ infrastructure in some way to help you. – Prateek Shrivastava Feb 13 '20 at 05:54
  • I would use a 2 table database based mechanism to do this. First table will hold the message from the Q. second table will hold the Okay to Process message. I will have one processes/thread scanning the MQ for messages. Pick the message and insert it into the relevant table. If it goes to first table respond with an ACK message. – Prateek Shrivastava Feb 13 '20 at 05:57
  • Another process/thread will keep looking at the Second Table for new records and process the corresponding message in FirstTable. If required write an ACK back to confirm message processed. Robustness wise - I would ideally Peek a message from Queue, identify which table it goes to, write to that table and then commit/tell the Q to remove the message. This way I am protected against in flight loss of messages. – Prateek Shrivastava Feb 13 '20 at 06:00
  • @PrateekShrivastava That was one of our original ideas, but as these tables grow in size the amount of time it will take to select data from them and then determine which table to write too/which record to update, we might be very close to the time restrictions we have imposed on us. We're trying to find a way to do it with as least amount of processing/wait time as possible. – Jimenemex Feb 13 '20 at 06:06
  • @JoshMc Those are the requires made by the client. They need acknowledgements at every step of way. I suppose the reason is because there are monetary transactions in the messages. – Jimenemex Feb 13 '20 at 13:18
  • Have the sending app send when it is ready to have the message processed. Recieving side app can send both an immediate ack and a response once processed. – JoshMc Feb 13 '20 at 13:45
  • Clearing Database is not a big task anyways. You can come up with policy for how much data to keep. End of Day a process can move data around. The design really depends on what business needs are. You can have new table per day (Date Stamped in table name). So every morning you start afresh on a blank table (benefits = no archiving process needed). There could be many ways. – Prateek Shrivastava Feb 13 '20 at 21:43
  • As you said this is monetary transaction processing system, I would now push towards Database table and logging. As these checkpoints can be your evidence if something goes wrong. You can evaluate a few NoSQL databases for performance reasons. – Prateek Shrivastava Feb 13 '20 at 21:45

1 Answers1

2

I'd do it with message expiry and activity reporting in MQ, but that doesn't make it unnecessary for your applications to have a log of what they sent and received.

On every message you send, you could set an expiry time (meaning that the message will be discarded, if it is removed from the queue outside of that time frame). And by also setting the reporting option for getting reports when the message is discarded, the sender of the message (actually you can specify any queue for the reports not just one monitored by the sender) will get a report message identifying the discarded message, which can then be resent.

However the discarded message won't be put to any queue, the sending application needs to be able to resend the message from scratch.

And the processing side, when it's waiting on the 2nd message, it will need to have it's own "timer" for handling timeouts.

https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.1.0/com.ibm.mq.ref.dev.doc/q097490_.htm

https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.1.0/com.ibm.mq.mon.doc/q036600_.htm

All that said, it seems to me that you are not using MQ as intended and are trying to mimic a synchronous communication. MQ can gurantee exactly once delivery, so I'd say that if you have a properly configured and sized MQ route between your applications and properly set up message monitoring, then all this ack/nack is unnecessary.

Attila Repasi
  • 1,803
  • 10
  • 11