Websphere 7 MQ Application stops picking up messages

Question

I have a websphere based piece of code that is designed to pickup messages off of a MQ Queue and process them. Most of the time the code works fine but every few days, the code continues to run however it stops picking up messages even though there are messages still on the queue. In order to get the process back working I need to bump the application and then everything starts working fine.

The messages can be quite large (up to 4MB per message), I am running on WAS 7

I don't see any messages or exceptions that seem to represent an error.

Here is the code for folks to comment on

public class BlipMqProcessor {

    protected static final int ONE_SECOND = 1000;
    protected static final int ONE_HOUR = 60 * 60 * ONE_SECOND;
    protected static final int MQ_READ_TIMEOUT = Integer.parseInt(Constants.MQ_READ_TIMEOUT_IN_SECONDS) * ONE_SECOND;

    protected static int previousMqReasonCode;
    protected static long previousMqErrorTime;

    private BlipXmlProcessor xmlProcessor;

    // Member variables for MQ processing
    protected MQQueueManager qMgr;
    protected MQQueue queue;
    protected MQGetMessageOptions gmo;

    /**
     * Constructs a new BlipMqProcessor with the given values.
     */
    public BlipMqProcessor() {
        this(new BlipXmlProcessor());
    }

    /**
     * Constructs a new BlipMqProcessor with the given values.
     * @param xmlProcessor the processor that will be used to create the
     *      staging table entries.
     */
    public BlipMqProcessor(final BlipXmlProcessor xmlProcessor) {
        super();
        this.xmlProcessor = xmlProcessor;
    }

    /**
     * Reads XML messages from the Constants.MQ_ACCESS_QUEUE_NAME
     * 
     * @throws BlipModelException if there are any 
     */
    public void readFromMQ() throws BlipModelException {
        try {
            createNewConnectionToMQ();
            while(true) {
                MQMessage outputMessage = new MQMessage();
                queue.get(outputMessage,gmo);
                String blipModelXml = outputMessage.readLine();
                BlipLogs.logXML("BlipREQ", "0", blipModelXml);
                processMessage(blipModelXml);
                qMgr.commit();
            }
        } catch (final MQException e) {
            if (e.reasonCode != MQException.MQRC_NO_MSG_AVAILABLE) {
                handleMqException(e);
            }
        } catch (final IOException e) {
            throw new BlipModelException("MQ", "Error reading MQ message.", "BlipMqProcessor.readFromMQ", e);
        } finally {
            cleanupMQResources();
        }
    }


    /**
     * Clean up MQ resources.
     */
    private void cleanupMQResources() {
        // Close queue
        if(queue != null) {
           try {
              queue.close();
           }catch(final MQException e) {
               BlipModelLogger.error("MQ", "BlipMqProcessor", "Problem closing queue: " + e);
           }
        }
        // Disconnect queue manager
        if(qMgr != null) {
            try {
                qMgr.disconnect();
            } catch (final MQException e) {
                BlipModelLogger.error("MQ", "BlipMqProcessor", "Problem disconnecting from qMgr: " + e);
            }
        }
    }

    protected void createNewConnectionToMQ() throws MQException {
        try {
            MQEnvironment.hostname = Constants.MQ_HOST;
            MQEnvironment.channel = Constants.MQ_CHANNEL;
            MQEnvironment.port      = Integer.parseInt(Constants.MQ_PORT);
            if(Constants.MQ_SSL_CIPHER_SUITE != null) {
                MQEnvironment.sslCipherSuite = Constants.MQ_SSL_CIPHER_SUITE;
                MQEnvironment.sslPeerName = Constants.MQ_SSL_PEER;
            } else {
                MQEnvironment.sslCipherSuite = "";
                MQEnvironment.sslPeerName = "";
            }

            qMgr = new MQQueueManager(Constants.MQ_QMGR);
            int openOptions = MQC.MQOO_INPUT_AS_Q_DEF;
            queue = qMgr.accessQueue(Constants.MQ_IN_ACCESS_QUEUE, openOptions);
            gmo = new MQGetMessageOptions();
            gmo.options = MQC.MQGMO_WAIT | MQC.MQGMO_SYNCPOINT | MQC.MQGMO_FAIL_IF_QUIESCING;
            gmo.waitInterval = MQ_READ_TIMEOUT;
        } finally {
            MQEnvironment.sslCipherSuite = "";
            MQEnvironment.sslPeerName = "";
        }
    }

    protected void handleMqException(final MQException e) {
        long currentTime = System.currentTimeMillis();
        long timeBetweenMqErrors = currentTime - previousMqErrorTime;
        if (previousMqReasonCode != e.reasonCode || timeBetweenMqErrors > ONE_HOUR) {
            previousMqReasonCode = e.reasonCode;
            previousMqErrorTime = currentTime;
            BlipModelLogger.error("MQ", "BlipMqProcessor", "MQException reading from Access Queue: " + e);
        }
    }


}

for the rest ... please to check [my question here](http://stackoverflow.com/questions/11056776/convert-string-from-ebcdic-to-unicode-utf8) — mKorbel, Jun 29 '12 at 16:53
Does your input queue have `BOQTHRESH` and `BOQNAME` set? If your app encounters a poison message and does NOT have a backout queue and backout threshold defined, it will back the message out until it exceeds the JEE container's backout threshold. At that point, the app stops getting new messages but since it does not see this as fatal, it looks happy. Some poison message errors are caused by tuning and thus clear on restart. It's a long shot but VERY easy to define a new queue and set `BOQNAME(NEW_QUEUE)` and `BOQTHRESH(5)` and every app should have these anyway. — T.Rob, Jul 01 '12 at 17:21

score 0 · Answer 1 · answered Jun 29 '12 at 19:31

Change the readFromMQ method;

public void readFromMQ() throws BlipModelException {
    try {
        createNewConnectionToMQ();
        while(true) {
          try {
            MQMessage outputMessage = new MQMessage();
            queue.get(outputMessage,gmo);
            String blipModelXml = outputMessage.readLine();
            BlipLogs.logXML("BlipREQ", "0", blipModelXml);
            processMessage(blipModelXml);
            qMgr.commit();
          } catch (MQException e) {
            if (e.reasonCode != MQException.MQRC_NO_MSG_AVAILABLE) {
              throw e;
            }
          }
        }
    } catch (final MQException e) {
      handleMqException(e);
    } catch (final IOException e) {
        throw new BlipModelException("MQ", "Error reading MQ message.", "BlipMqProcessor.readFromMQ", e);
    } finally {
        cleanupMQResources();
    }
}

Will be the quick solution; but not elegant. There's plenty of scope for refactoring here.

What's happening is that you are in fact getting a MQRC_NO_MSG_AVAILABLE because funnily enough, there isn't another message available to retrieve (at some point in time, not when you think); by the time you've decided to ignore that exception you're already exited out of the while(true) loop.

You can't use queue.getCurrentQueueDepth() because it's not performant (apparently) and also because it won't work with alias queues or queues living on a cluster. This is pretty much it; it sucks.

When you open a queue, WMQ returns the resolved name so you have something you can inquire on the depth of should you wish to do so. Using `queue.getCurrentQueueDepth()` is not performant *if you only care whether the queue depth is > 0 because you want to get the next message*. Executing a `GET` and receiving either a message or MQRC_NO_MSG_AVAILABLE is always more performant than inquiring on depth and then executing a GET if depth < 0 (and eliminates a race condition). Inquiring the depth of a queue for which you already have an open handle is quite fast. — T.Rob, Jul 01 '12 at 17:33

Websphere 7 MQ Application stops picking up messages

1 Answers1