Too many mq connections for a server connection channel which are not created by queue

Question

As per our configuration, we have WAS version is 8.5.5.11, IBM MQ version 7.5.0.3. We are using 2 channels to connect to WMQ, one with MAXINST set to 250 and one with 500. SHARECNV is set to 10 for both. Now we have an upper limit of making maximum 2000 connections in a queue manager but we end up crossing that limit after 3-4 days of continuous running of WAS Server. After doing some analysis we can see that at any point of time we have only 120-160 active connections. DIS CONN command gives shows a lot of connections with OBJNAME, OBJTYPE empty and ASTATE "NONE". Even these connections are made from our WAS Server IP but it seems they are not created by queues as OBJTYPE is not Queue for these connections. These connections keep on growing over a period of time and ultimately we reach our limit of 2000 connections.

Can somebody help in identifying why these connections are getting created and make sure they get closed like normal idle connections. This is how connection are made and closed in the application: We have an abstrack bean class which is extended by all MDB's.

@MessageDriven(activationConfig = {
@ActivationConfigProperty(propertyName = "acknowledgeMode", propertyValue = "Auto-acknowledge"),
@ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue") })
public class TrackBeanV2 extends AbstractServiceBean implements MessageListener {//code}

The abstrack bean handles creation and closing of connections in following manner:

public abstract class AbstractServiceBean {
@Resource(name = "myQCF", type = QueueConnectionFactory.class, shareable =     true, description = "Reply Connection Factory")
private ConnectionFactory replyCF; 

@PostConstruct
private void postConstruct() {
    replyConnection = replyCF.createConnection();

}  catch (JMSException e) {
    throw new RuntimeException("Failed to create JMS Connection");
}

}
@PreDestroy
private void preDestroy() {
try {
    replyConnection.close();
} catch (JMSException e) {
    throw new RuntimeException("Failed to close JMS connection", e);
}
}

private void sendResponseMessage(String outputMessageText, String  jmsMessageID , Destination replyDestination) {
TextMessage replyMessage = null;
try {           
    createSession();    
    createProducer();
    replyMessage = createReplyMessage(outputMessageText , jmsMessageID);    
    sendReply(replyMessage, replyDestination);  
    closeProducer();
    closeSession();
} catch (JMSException exp) {
    handleException(exp);
}
}
private void createSession() throws JMSException{
replySession = replyConnection.createSession(true, 0);                  
}`
private void createProducer() throws JMSException{                              
replyProducer = replySession.createProducer(null);      
}

private void closeSession() throws JMSException {
if (replySession != null) {
    replySession.close();
}
}

private void closeProducer() throws JMSException{
if (replyProducer != null) {            
    replyProducer.close();          
}
}   
private void sendReply(TextMessage replyMessage, Destination replyDestination) throws JMSException {    
logMessages(replyMessage.getText(), "RESPONSE MESSAGE");
replyProducer.send(replyDestination, replyMessage);
}

An output of below command

echo "DIS CONN(*) TYPE(*) CONNAME CHANNEL OBJNAME OBJTYPE" | mqsc -e -m QPDC1GC2 -p width=1000 | grep -o '^\w\+:\|\w\+[(][^)]\+[)]' | awk -F '[()]' -v OFS="," 'function printValues() { if ("CHANNEL" in p) { print p["CHANNEL"], p["CURSHCNV"], p["CONNAME"],p["CHSTADA"],p["CHSTATI"],p["LSTMSGDA"],p["LSTMSGTI"],p["OBJNAME"],p["OBJTYPE"],p["ASTATE"],p["APPLDESC"],p["APPLTAG"] } } /^\w+:/ { printValues(); delete p; next } { p[$1] = $2 } END { printValues() }' | grep MYCHANNEL

looks like this

A.QMGR1.MYCHANNEL,,10.217.278.15,,,,,VALIDATE_GET_01,QUEUE,ACTIVE,WebSphere MQ Channel,WebSphere MQ Client for Java
A.QMGR1.MYCHANNEL,,10.217.278.15,,,,,,,NONE,WebSphere MQ Channel,WebSphere MQ Client for Java
A.QMGR1.MYCHANNEL,,10.217.278.15,,,,,,,NONE,WebSphere MQ Channel,WebSphere MQ Client for Java
A.QMGR1.MYCHANNEL,,10.217.278.15,,,,,,,NONE,WebSphere MQ Channel,WebSphere MQ Client for Java
A.QMGR1.MYCHANNEL,,10.217.278.15,,,,,,,NONE,WebSphere MQ Channel,WebSphere MQ Client for Java
A.QMGR1.MYCHANNEL,,10.217.278.15,,,,,,,NONE,WebSphere MQ Channel,WebSphere MQ Client for Java
A.QMGR1.MYCHANNEL,,10.217.278.15,,,,,VALIDATE_GET_01,QUEUE,ACTIVE,WebSphere MQ Channel,WebSphere MQ Client for Java
A.QMGR1.MYCHANNEL,,10.217.278.15,,,,,GETDETAILSGET_01,QUEUE,NONE,WebSphere MQ Channel,WebSphere MQ Client for Java

This is Config in qm.ini.

Channels:
MaxChannels=2000
MaxActiveChannels=2000

Output of the command provided in the comments:

   "841DD95801B5DC20","A.QMGR1.MYCHANNEL","10.217.278.15","GETDETAILSGET_01","QUEUE","ACTIVE","WebSphere MQ Channel","WebSphere MQ Client for Java","MQOO_INPUT_SHARED,MQOO_BROWSE,MQOO_INQUIRE,MQOO_SAVE_ALL_CONTEXT,MQOO_FAIL_IF_QUIESCING"
   "841DD958AB2CF820","A.QMGR1.MYCHANNEL","10.217.278.15","GETDETAILSGET_01","QUEUE","NONE","WebSphere MQ Channel","WebSphere MQ Client for Java","MQOO_INPUT_SHARED,MQOO_INQUIRE,MQOO_SAVE_ALL_CONTEXT,MQOO_FAIL_IF_QUIESCING,MQOO_NO_READ_AHEAD"

Only a instance of a channel counts against the MaxChannels value set in qm.ini, you stated this is 2000. Since your two channels have a combined MAXINST of 750, those two channels by themselves can not be crossing your MaxChannels setting of 2000. You can have up to 10 conversations per channel instance so that gives you a max across those two channels of 7500 conversations. If the channel has 1 conversation or 10 conversations it still counts as 1 channel instance against MaxChannels. Can you try and clarify what limit you are reaching? — JoshMc, Mar 28 '17 at 14:50
@JoshMc Even I am not sure why we start facing issue with no of connections crossing 2000 and not the instances.. No of instances remain around 300 only. Still we are not able to make any new connections. — Neel, Mar 28 '17 at 15:05
And We can see that all 2000 connections are created by our WAS server IP, so other applications are not creating these connections — Neel, Mar 28 '17 at 15:06
Can you clarify when you state 2000 connections, how are you obtaining this? When you say number of instances, how are you obtaining this? Using the "DIS CHS" command I gave you in a prior question can you total the number of shared conversations to see if this matches the number of connections? You need a different awk command to parse the output of DIS CONN if you want to get OBJNAME and OBJTYPE. Please see my next comment for a DIS CONN command that will show OBJNAME and OBJTYPE if present. — JoshMc, Mar 29 '17 at 00:03
Try this command `echo "dis conn(*) type(all) where(channel eq A.QMGR1.MYCHANNEL)"|mqsc -e -m QPDC1GC2 -p width=1000|grep -o '^\w\+:\|\w\+[(][^)]\+[)]' | awk -F '[()]' -v OFS='","' 'function printValues() { if ("CONN" in p) { print p["CONN"], p["CHANNEL"], p["CONNAME"], p["OBJNAME"], p["OBJTYPE"], p["ASTATE"], p["APPLDESC"], p["APPLTAG"], p["OPENOPTS"] } } /^\w+:/ { if (x !~ /YES/) {printValues()}; x = "NO"; delete p; next } { p[$1] = $2 } { if ("OPENOPTS" in p) { printValues() ; delete p["OPENOPTS"]; x = "YES"} } END { if (x !~ /YES/) {printValues()} }'|sed -e 's/^/"/g' -e 's/$/"/g'` — JoshMc, Mar 29 '17 at 00:04
Note that not every connection has to have a object open, and a single connection can have multiple objects open. The long command I provided will display at least one row per connection even if no objects are open, this will have OBJNAME and OBJTYPE blank. It the conn had at least 1 object open it will display one row per object. — JoshMc, Mar 29 '17 at 00:04
@JoshMc Yes i counted the instances using DIS CHS command and as you mentioned they cannot go beyond 750, but the strange thing is we face issue when count of DIS CONN reaches around 2000 and not CHS. I ran the dis conn command u provided also and there also i can see lot of connections with OBJNAME and OBJTYPE blank. I don't see such connections in other channels in the other queue managers we have. When are these connections created? Is this is a normal behaviour? Also it seems they are long running and don't close as per idle timeout set in the queue connection factory which is 10 mins. — Neel, Mar 29 '17 at 08:00
@JoshMc I have added the output of the command in the question. Currently there are 110 connections out of which 102 are with ASTATE NONE and no OBJTYPE or OBJNAME. 5 are active and 2 are with ASTATE NONE but with OBJTYPE and OBJNAME present. This does seem to be correct behavior — Neel, Mar 29 '17 at 08:28
There is a limit for connection from the same client called MAXINSTC. What do you have that set to? — Attila Repasi, Mar 29 '17 at 11:28
@AttilaRepasi We have not put limit on client, it is default 999999999. — Neel, Mar 29 '17 at 11:48
What is the actual error message you get, is there an MQ RC? Can you check the error log of the QM for any related error? — Attila Repasi, Mar 29 '17 at 12:05
@AttilaRepasi We do not get any error in MQ side.. we start getting error in WAS side which are timeout exceptions waiting for connections which resolve after a WAS server restart — Neel, Mar 29 '17 at 12:38

score 1 · Answer 1 · answered Apr 02 '17 at 09:43

The symptoms point to your application connections not getting closed properly. I have a IBM MQ administrator background not a Java development background, but based on what you have posted so far it appears to me that you are attempting to close the connections properly. Hopefully someone with more IBM MQ classes for JMS development background can take a look. @Roger?

The following script will display the number of channel instances for a specific channel name. It will also show the total shared conversations across that channel's instances, the number of unique CONNs, and the total number of objects opened by the CONNs including CONNs with no objects open. Note that when a IBM MQ client process connects to the queue manager a unique CONN to the queue manager will be created, if the process opens a object such as the queue manager, a queue, or a topic, DIS CONN will display those open objects as well.

This may help to observer the increase in these numbers over time to help you understand the connection/channel leak you are experiencing.

Example output:

./mq-info.ksh MQ.CHANNEL.NAME QMGR.NAME
Total Number of channel instances: 53
Total shared conversations: 94
Unique CONNs: 94
Objects opened by CONNs:
     15 ""
     45 "QMGR"
    120 "QUEUE"

Note that if the channel is configured with SHARECNV of at least 1, the value of Total shared conversations and Unique CONNs should always match.

In the example output 15 of the shared conversations have no objects open. The remaining 70 shared conversations have the queue manager itself open 45 times and queue objects open 120 times.

mq-info.ksh is below:

#!/bin/ksh

CHL=${1}
QMGR=${2}
echo "Total Number of channel instances: $(echo "DIS CHS(${CHL}) ALL"|runmqsc $QMGR| grep -o '^\w\+:\|\w\+[(][^)]\+[)]' | awk -F '[()]' -v OFS="," 'function printValues() { if ("CHANNEL" in p) { print p["CHANNEL"], p["CURSHCNV"], p["CONNAME"] } } /^\w+:/ { printValues(); delete p; next } { p[$1] = $2 } END { printValues() }'|wc -l)"
echo "Total shared conversations: $(expr $(echo $(echo "DIS CHS(${CHL}) ALL"|runmqsc $QMGR| grep -o '^\w\+:\|\w\+[(][^)]\+[)]' | awk -F '[()]' -v OFS="," 'function printValues() { if ("CHANNEL" in p) { print p["CHANNEL"], p["CURSHCNV"], p["CONNAME"] } } /^\w+:/ { printValues(); delete p; next } { p[$1] = $2 } END { printValues() }'|awk -F, '{print $2}')|sed -e 's/ / + /g'))"
echo "Unique CONNs: $(echo "dis conn(*) type(all) where(channel eq ${CHL})"|runmqsc $QMGR|grep -o '^\w\+:\|\w\+[(][^)]\+[)]' | awk -F '[()]' -v OFS='","' 'function printValues() { if ("CONN" in p) { print p["CONN"], p["CHANNEL"], p["CONNAME"], p["OBJNAME"], p["OBJTYPE"], p["ASTATE"], p["APPLDESC"], p["APPLTAG"], p["OPENOPTS"] } } /^\w+:/ { if (x !~ /YES/) {printValues()}; x = "NO"; delete p; next } { p[$1] = $2 } { if ("OPENOPTS" in p) { printValues() ; delete p["OPENOPTS"]; x = "YES"} } END { if (x !~ /YES/) {printValues()} }'|sed -e 's/^/"/g' -e 's/$/"/g'|awk -F, '{print $1}'|sort -u|wc -l)"
echo "Objects opened by CONNs:"
echo "dis conn(*) type(all) where(channel eq ${CHL})"|runmqsc $QMGR|grep -o '^\w\+:\|\w\+[(][^)]\+[)]' | awk -F '[()]' -v OFS='","' 'function printValues() { if ("CONN" in p) { print p["CONN"], p["CHANNEL"], p["CONNAME"], p["OBJNAME"], p["OBJTYPE"], p["ASTATE"], p["APPLDESC"], p["APPLTAG"], p["OPENOPTS"] } } /^\w+:/ { if (x !~ /YES/) {printValues()}; x = "NO"; delete p; next } { p[$1] = $2 } { if ("OPENOPTS" in p) { printValues() ; delete p["OPENOPTS"]; x = "YES"} } END { if (x !~ /YES/) {printValues()} }'|sed -e 's/^/"/g' -e 's/$/"/g'|awk -F, '{print $5}'|sort|uniq -c

Thanks for sharing the script. The output for both the channels looks like this Total Number of channel instances: 106 Total shared conversations: 392 Unique CONNs: 392 Objects opened by CONNs: 110 "" 220 "QMGR" 62 "QUEUE" --------------------------- Total Number of channel instances: 33 Total shared conversations: 246 Unique CONNs: 246 Objects opened by CONNs: 94 "" 138 "QMGR" 14 "QUEUE" — Neel, Apr 02 '17 at 12:51
For our channels SHARECNV is set to 10 but still Total shared conversations and Unique CONNs value is same. — Neel, Apr 02 '17 at 13:09
@Neel That would be correct, if SHARECNV at least 1 -- it can be higher than one, if you set it to 0, then the channel works in pre-v7 mode and does not support shared conversations and the value for CURSHCNV in CHS will show 0 so the "Total shared conversations" would show 0. This is also the case if your app has jar files from a pre-v7 version of MQ no atter what SHARECNV is set to. — JoshMc, Apr 02 '17 at 16:42
But in my case CURSHCNV is not zero.. It has values ranging from 1 to 10.What about so many QMGR OBJTYPE connections. I see them growing over past 3 hours. Seems like they are not getting closed. — Neel, Apr 02 '17 at 16:51
@Neel I was just noting a different behavior for anyone else who may be using SHARECNV(0) or a pre-v7 client jar file. This does not apply to you. Any value 1 or higher would have the output you described which is expected that shared conversations would be equal to unique CONNs. — JoshMc, Apr 02 '17 at 20:59
@Neel I agree that it does seem like they are not getting closed and that was the first statement I made in the answer to this question and also in your prior question. I don't know why they do not close, someone with more Java/JMS experience would need to weigh in. — JoshMc, Apr 02 '17 at 21:00
Some JMS connections open the QMGR object to inspect the DEADQ parameter. Not all do. A connection does not have to have any object open which is reflected in the count with "". — JoshMc, Apr 02 '17 at 21:01
Issue definitey seems to be due to JMS connections which have opened QMGR object as after a WAS server restart QMGR connections became 0, rest are almost the same. — Neel, Apr 03 '17 at 08:34
it is created in following manner. We are not destroying it anywhere jmsMessageID = message.getJMSMessageID(); replyDestination = message.getJMSReplyTo(); — Neel, Apr 04 '17 at 08:43
Can you take a look at /opt/mqm/samp/jms/samples/JmsProducer.java and try doing something similar to wrap your `.close` calls in a try/catch block and logging any JMS error AND linked exception, in the sample this is via recordFailure and processJMSException. — JoshMc, Apr 04 '17 at 09:00
@Neel, I noted that you found a resolution in some posts you made on another site, you should post a answer with the resolution to this problem. — JoshMc, Jun 20 '17 at 10:28

Too many mq connections for a server connection channel which are not created by queue

1 Answers1