3

We have a very simple application which read messages from MQ and transform the messages into some different format and send it to another MQ. The application is deployed in the Openshift environment. Hence at any given time multiple instances of the same application will be running. We are not using any DB in our application.

We want to maintain the order of messages. Basically while reading the messages, they were present in source MQ with some order(for e.g. A1-A2-A3-A4-A5) then post transformation they should be in same order in out MQ.

Eg.: Let's suppose event A1 has happened then A2 and then A3 and then A4 and then A5. Events A1 and A3 are interlinked means has some relation. Multiple instances of same application are listening on dedicated Queue for these events. Let's suppose there were 5 instances of applications running, then there can be a case where instance which has picked A3 event has finished processing. While A1 event processing still not completed. So basically I want to ensure A3 processed output should not be sent to out Queue until its previous event A1 is completed successfully and sent to out queue.

Can someone please help me with this problem?

Daniel Pop
  • 456
  • 1
  • 6
  • 23
vivek075
  • 71
  • 6
  • What behaviour would you expect if the second job receives A2…A5 but not A1 (due to common failures like network issues) – Abhinav Mathur Jul 11 '22 at 17:37
  • The best I see is that you can split messages into groups and have each group arrive in logical order. Groups can still be out of order relative to each other though. See https://www.ibm.com/docs/en/ibm-mq/9.1?topic=queue-logical-physical-ordering for more. – btilly Jul 11 '22 at 21:38
  • do the messages belong to some kind of session? What I mean by that is if there is a way to logically group a batch of messages together OR the all need to be processed sequentially – David Guida Jul 12 '22 at 11:17
  • If there are different types of messages on the same queue and order only matters for each type of message then one solution could be to have an app that splits the messages across multiple queues and then you can run multiple instances of your transformation app. If the messages could have a message property added at the source then you could even run multiple listeners each with a selector on the single input queue. – JoshMc Jul 12 '22 at 17:57
  • 1
    Strictly if you have more than one application getting messages then you will have messages out of order. For example Instance1 gets message, Puts reply, Instance2 gets message puts reply. instance 2 commits, instance 1 commits. For a short while, reply2 was available, before reply1 was committed. A stock exchange needs to process stock orders for a company in strict order. They solved this by having multiple queues. QUEUE A_B for all stocks beginning with A or B, QUEUE C for all stocks beginning with C... V-Z... etc. For each queue they had 1 app processing the queue. – colin paice Jul 13 '22 at 10:20
  • @colinpaice Then there will be lot of queues right? If suppose there are 10K companies stocks trading then theoretically 10K queues will be created, and queues keep on increasing with every new company? – vivek075 Jul 13 '22 at 14:44
  • @JoshMc Generally messages will be different; only when there is some change in state of previous message received; we will get an amended message. Basically I am looking for way where in Amended message should not be processed before its previous version. Hope you have got what I am looking for. Will refine Question for better clarity. – vivek075 Jul 13 '22 at 14:50
  • You don't need 10k queues, you can split them how you want as long as they are always split consistently. I know one app that created a hash based on some field of the message and had 20 readers, each reader would only process a subset of hashes, but in total all hashes were processed. – JoshMc Jul 13 '22 at 15:40
  • You can have as many of few queues as you need. If one cant handle it, try 10 - but have a consistent way of allocating messages to queues. Eg hash (from Josh),could be first letter of an attribute or all A* B* go to queue A_B. I picture a big switch statement. – colin paice Jul 14 '22 at 16:39

2 Answers2

0

IBM MQ has built-in features for grouping messages together. I would suggest you have a look at Message Groups.

Roger
  • 7,062
  • 13
  • 20
  • Can you please help me understand how it will solve the problem? – vivek075 Jul 13 '22 at 15:05
  • I gave you the link, just read up on Message Groups. When an instance starts getting messages for a group, then no other instances will get any of those messages in the group. – Roger Jul 13 '22 at 23:45
0

When consuming messages store in a file according to the classification. When a message consumed and decided to a group according to the classification. Update the file. It can be done by the first answer for first way you have to specify a message a group ID on program and it will process messages as a group.