4

I have configured CachingConnectionFactory that wraps a MQTopicConnectionFactory and MQQueueConnectionFactory with cache size set to 10 each.

These are than used in several jms:outbound-channel-adapter or jms:message-driven-channel-adapter as part of various spring integration workflows that I have in my application.

It is noticed that once in a while the connection count on MQ channel reaches maximum allowed (about 1000) when the process stops functioning. This is a serious problem for a production application.

Bringing the application down does not reduce the connection count so looks like orphaned connections on MQ side? I am not sure if I am missing anything in my spring jms / SI configuration that can resolve this issue, any help would be highly appreciated.

Also I would like to log connection open and close from application but don't see a way to do that.

<bean id="mqQcf" class="com.ibm.mq.jms.MQQueueConnectionFactory">
//all that it needs host/port/ queue manager /channel
</bean>

 <bean id="qcf" class="org.springframework.jms.connection.CachingConnectionFactory">
            <property name="targetConnectionFactory" ref=" mqQcf "/>
            <property name="sessionCacheSize" value="10"/>          
</bean>


<bean id="mqTcf" class="com.ibm.mq.jms.MQTopicConnectionFactory">
//all that it needs host/port/ queue manager /channel
</bean>

<bean id="tcf" class="org.springframework.jms.connection.CachingConnectionFactory">
            <property name="targetConnectionFactory" ref=" mqTcf "/>
            <property name="sessionCacheSize" value="10"/>          
</bean>

//Qcf and tcf are than used in spring integration configuration as required
JoshMc
  • 10,239
  • 2
  • 19
  • 38
Mark1234
  • 589
  • 2
  • 8
  • 24

2 Answers2

1

You really need to show your configuration but the Spring CachingConnectionFactory only creates a single connection that's shared for all sessions. Turning on INFO logging for the CCF category emits this log when a new connection is created...

if (logger.isInfoEnabled()) {
    logger.info("Established shared JMS Connection: " + this.target);
}

EDIT:

There's nothing in your config that stands out. As I said, each CCF will have at most 1 connection open at a time.

One possibility, if you have idle times, is that the network (a switch or firewall) might be silently dropping connections without telling the client or server. The next time the client tries to use its connection it will fail and create a new one but the server may never find out that the old one is dead.

Typically, for such situations, enabling heartbeats or keepalives would keep the connection active (or at least allow the server to know it's dead).

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • Just added some code sample to the question, this is nothing special, just the very basic required configuration for connection factory, let me know if you would like to see any other code, thanks! Also i did enable logging for above and see a shared connection being printed, but it has to create more connections to serve various publishers and subscribers, also when concurrent consumers are set to higher numbers? – Mark1234 Jan 05 '15 at 21:22
  • Thanks Gary, is there a way to tell CachingConnectionFactory to close a connection if not used for some time? – Mark1234 Jan 05 '15 at 22:08
  • 1
    Not built it; it does have a method `resetConnection()` that you can call, but it would pull the rug on any in-process activities. – Gary Russell Jan 05 '15 at 22:27
  • And it wouldn't help if the network already pulled the plug on the connection; heartbeats/keepalives are the only reliable mechanism. – Gary Russell Jan 05 '15 at 22:52
  • On a related note, i notice that the logs prints that ..Established shared JMS Connection: com.ibm.mq.jms.MQQueueConnection....just once but the connection count on MQ goes up by 2, any reason why this is the case? Thanks – Mark1234 Jan 07 '15 at 23:30
  • No idea; I suggest you check with netstat and/or wireshark or similar but the connection factory only opens one connection. – Gary Russell Jan 07 '15 at 23:44
  • I am seeing another strange thing while publishing to a topic, these are three lines o.s.i.jms.JmsSendingMessageHandler - org.springframework.integration.jms.JmsSendingMessageHandler#1 received message: [Payload String ...] o.s.j.c.CachingConnectionFactory - Established shared JMS Connection: com.ibm.mq.jms.MQTopicConnection@246adb31 o.s.j.c.CachingConnectionFactory - Registering cached JMS Session for mode 1: com.ibm.mq.jms.MQTopicSession@473eae6e o.s.i.jms.DynamicJmsTemplate - Executing callback on JMS Session: Cached JMS Session: com.ibm.mq.jms.MQTopicSession@473eae6e ..than stuck, for ever – Mark1234 Jan 08 '15 at 16:38
  • Also can i add some timeout to a jms:outbound-channel-adapter so it does not hang forever for unknown reason. – Mark1234 Jan 08 '15 at 16:49
  • Not if it's hung in the MQ client library (unless it has some settings); I suggest you take a thread dump with jstack to see where the thread is. In future, it's easier if you edit your question rather than putting code/logs in comments. – Gary Russell Jan 08 '15 at 17:25
0

I was debugging a similar issue in my application about the number of open output counts in MQ when there is only one Connection is opened by the connection factory.

The number output counts in MQ explorer is the number of connection handles created by the IBM MQ classes. Per IBM documentation, A Session object encapsulates an IBM MQ connection handle, which therefore defines the transnational scope of the session.

Since the session cache size was 10 in my application, there were 10 IBM MQ connections handles created (one for each session) stayed open for days and the handle state was inactive.

More info can be found in,

https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.0.0/com.ibm.mq.dev.doc/q031960_.htm

As Gary Russell mentioned, Spring doesn't provide way to configure the time outs for these idle connections. IBM has inbuilt properties in MQConnectionFactory which can be configured to setup the reconnect timeouts.

More infor can be found in,

https://www.ibm.com/developerworks/community/blogs/messaging/entry/simplify_your_wmq_jms_client_with_automatic_client_reconnection19?lang=en

The reconnect on exception is true by default for CCF. So care should be taken if IBM throws an exception after time out interval. I am not sure if there is a max number of times reconnect will try before throwing an exception in CCF.

Selvakumar
  • 189
  • 1
  • 13
  • The CCF maintains a cache for producer too inside each session. On further debugging of the connection issues in my application, the total IBM connection handles created and stays idle is = session cache size * number of unique destinations. There is no guarantee that you will get the same session again from the cache next time and you end up using the producer from cache. So this is a overhead to the application and connections stays idle for days without re-use. So care should be taken while deciding on the session cache size and also whether to use producer cache. – Selvakumar Jul 07 '17 at 20:34