how to read large volume of messages from Websphere MQ

Question

I want to read 10000 messages from Websphere MQ in groups in sequential order, i am using below code to do the same, but it is taking long time to read all the messages. Even i tried to use multi thread concepts, but sometimes 2 threads are consuming same group and race condition happening. Below is the code snippet. I am trying to use 3 threads to read 10000 messages from MQ sequentially, but two of my threads accessing same group at time. How to avoid this ? what is best way to read large volume of messages in sequential.? My requirement is i want to read 10000 messages sequentially. Please help.

MQConnectionFactory factory = new MQConnectionFactory();
factory.setQueueManager("QM_host")
MQQueue destination = new MQQueue("default");
Connection connection = factory.createConnection();
connection.start();
Session session = connection.createSession(true, Session.AUTO_ACKNOWLEDGE);

MessageConsumer lastMessageConsumer = 
    session.createConsumer(destination, "JMS_IBM_Last_Msg_In_Group=TRUE");
TextMessage lastMessage = (TextMessage) lastMessageConsumer.receiveNoWait();
lastMessageConsumer.close();

if (lastMessage != null) {

    int groupSize = lastMessage.getIntProperty("JMSXGroupSeq");
    String groupId = lastMessage.getStringProperty("JMSXGroupID");

    boolean failed = false;

    for (int i = 1; (i < groupSize) && !failed; i++) {

        MessageConsumer consumer = session.createConsumer(destination,
            "JMSXGroupID='" + groupId + "'AND JMSXGroupSeq=" + i);
        TextMessage message = (TextMessage)consumer.receiveNoWait();

        if (message != null) {
            System.out.println(message.getText());
        } else {
            failed = true;
        }

        consumer.close();

    }

    if (failed) {
        session.rollback();
    } else {
        System.out.println(lastMessage.getText());
        session.commit();
    }

}

connection.close();

Answer 1

I think a better way would be to have a coordinator thread in your application, which would listen for the last messages of groups and for each would start a new thread to get messages belonging in the group assigned to that thread. (This would cater for the race conditions.)

Within the threads getting the messages belonging in a group, you don't need to use a for loop to get each message separately, instead you should take any message belonging in the group, while maintain a group counter and buffering out of order messages. This would be safe as long as you commit your session only after receiving and processing all messages of the group. (This would yield more performance, as each group would be processed by a separate thread, and that thread would only access every message once in MQ.)

Answer 2

Please see IBM's documentation on sequential retrieval of messages. In case the page moves or is changed, I'll quote the most relevant part. For sequential processing to be guaranteed, the following conditions must be met:

• All the put requests were done from the same application.
• All the put requests were either from the same unit of work, or all the put requests were made outside of a unit of work.
• The messages all have the same priority.
• The messages all have the same persistence.
• For remote queuing, the configuration is such that there can only be one path from the application making the put request, through its queue manager, through intercommunication, to the destination queue manager and the target queue.
• The messages are not put to a dead-letter queue (for example, if a queue is temporarily full).
• The application getting the message does not deliberately change the order of retrieval, for example by specifying a particular MsgId or CorrelId or by using message priorities.
• Only one application is doing get operations to retrieve the messages from the destination queue. If there is more than one application, these applications must be designed to get all the messages in each sequence put by a sending application.

Though the page does not state this explicitly, when they say "one application" what is meant is a single thread of that one application. If an application has concurrent threads, the order of processing is not guaranteed.

Furthermore, reading 10,000 messages in a single unit of work as suggested in another response is not recommended as a means to preserve message order! Only do that if the 10,000 messages must succeed or fail as an atomic unit, which has nothing to do with whether they were received in order. In the event that large numbers of messages must be processed in a single unit of work it is absolutely necessary to tune the size of the log files, and quite possibly a few other parameters. Preserving sequence order is torture enough for any threaded async messaging transport without also introducing massive transactions that run for very long periods of time.

Answer 3

You can do what you want with MQ classes for Java (non-JMS) and it may be possible with MQ classes for JMS but be really tricky.

First read this page from the MQ Knowledge.

I converted the pseudo code (from the web page above) to MQ classes for Java and changed it from a browse to a destructive get.

Also, I prefer to do each group of messages under a syncpoint (assuming a reasonable sized groups).

First off, you are missing several flags for the 'options' field of GMO (GetMessageOptions) and the MatchOptions field needs to be set to 'MQMO_MATCH_MSG_SEQ_NUMBER', so that all threads will always grab the first message in the group for the first message. i.e. not grab the 2nd message in the group for the first message as you stated above.

MQGetMessageOptions gmo = new MQGetMessageOptions();
MQMessage rcvMsg = new MQMessage();

/* Get the first message in a group, or a message not in a group */
gmo.Options = CMQC.MQGMO_COMPLETE_MSG | CMQC.MQGMO_LOGICAL_ORDER | CMQC.MQGMO_ALL_MSGS_AVAILABLE | CMQC.MQGMO_WAIT | CMQC.MQGMO_SYNCPOINT;
gmo.MatchOptions = CMQC.MQMO_MATCH_MSG_SEQ_NUMBER;
rcvMsg.messageSequenceNumber = 1;
inQ.get(rcvMsg, gmo);

/* Examine first or only message */
 ...

gmo.Options = CMQC.MQGMO_COMPLETE_MSG | CMQC.MQGMO_LOGICAL_ORDER | CMQC.MQGMO_SYNCPOINT;
do while ((rcvMsg.messageFlags & CMQC.MQMF_MSG_IN_GROUP) == CMQC.MQMF_MSG_IN_GROUP)
{
    rcvMsg.clearMessage();
    inQ.get(rcvMsg, gmo);
   /* Examine each remaining message in the group */
   ...
}
qMgr.commit();