We are using Spring's CachingConnectionFactory on the sender side of JMS application which sends messages to Tibco EMS.
We are getting problems when EMS broker is being restarted or there is a fault-tolerant switch.
The call which handles exception handling in CachingConnectionFactory hangs and it keeps lock which prevent any future client calls to EMS from being processed.
Here is the stack trace
TIBCO EMS TCPLink Reader (Server-880629)" - Thread t@457
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Native Method)
- waiting on <20035387> (a com.tibco.tibjms.TibjmsxResponse)
at java.lang.Object.wait(Object.java:502)
at com.tibco.tibjms.TibjmsxLink.sendRequest(TibjmsxLink.java:359)
at com.tibco.tibjms.TibjmsxLink.sendRequestMsg(TibjmsxLink.java:293)
at com.tibco.tibjms.TibjmsxSessionImp._close(TibjmsxSessionImp.java:2836)
at com.tibco.tibjms.TibjmsxSessionImp.close(TibjmsxSessionImp.java:4800)
at org.springframework.jms.connection.CachingConnectionFactory$CachedSessionInvocationHandler.physicalClose(CachingConnectionFactory.java:487)
at org.springframework.jms.connection.CachingConnectionFactory$CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:317)
at com.sun.proxy.$Proxy263.close(Unknown Source)
at org.springframework.jms.connection.CachingConnectionFactory.resetConnection(CachingConnectionFactory.java:184)
- locked <37550f2a> (a java.util.LinkedList)
- locked <7e847e44> (a java.util.HashMap)
at org.springframework.jms.connection.SingleConnectionFactory.onException(SingleConnectionFactory.java:360)
at org.springframework.jms.connection.SingleConnectionFactory$AggregatedExceptionListener.onException(SingleConnectionFactory.java:716)
at com.tibco.tibjms.TibjmsConnection._invokeOnExceptionCallback(TibjmsConnection.java:2029)
at com.tibco.tibjms.TibjmsConnection._reconnect(TibjmsConnection.java:1902)
at com.tibco.tibjms.TibjmsConnection$ServerLinkEventHandler.onEventReconnect(TibjmsConnection.java:364)
at com.tibco.tibjms.TibjmsxLinkTcp._doReconnect(TibjmsxLinkTcp.java:602)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:319)
- locked <405e282> (a java.lang.Object)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
There are essentially two locks. The first one ( represented as locked <405e282> in the stack trace above ) is shared with other threads and prevents any thread for sending new request
synchronized (TibjmsxLinkTcp.this._sendLock)
/* */ {
/* 315 */ if ((TibjmsxLinkTcp.this._state == 1) &&
/* */
/* 317 */ (TibjmsxLinkTcp.this._doReconnect(false))) {
/* 318 */ break label0:
/* */ }
/* */ }
which seems reasonable enough. The problem is with another lock, represented as <20035387> in the stack trace
try
/* */ {
/* 335 */ send(paramObject, i, localTibjmsxResponse);
/* */ }
…
synchronized (localTibjmsxResponse)
/* */ {
/* 351 */ while (localTibjmsxResponse.status == 0)
/* */
/* */
/* */ {
/* */ try
/* */ {
/* 357 */ if (paramLong < -1L)
/* 358 */ localTibjmsxResponse.wait(-paramLong);
/* */ else localTibjmsxResponse.wait();
/* */
/* */ }
/* */ catch (InterruptedException localInterruptedException)
/* */ {
/* 364 */ ??? = Tibjmsx.buildException("InterruptedException has occurred while waiting for server response", null, localInterruptedException);
/* */ }
/* */
/* */ }
The flag on localTibjmsxResponse is never changed so the thread hangs.
I would imagine that using Spring's CachingConnectionFactory with EMS is fairly common, wonder if anybody else run into the same issues and knows what the solution is ?
Here is how we create the configuration
public ConnectionFactory jmsConnectionFactory() {
if (logger.isInfoEnabled()) {
logger.info(
new StringBuilder("Initialising JMSSender using providerUrl=").append(this.providerUrl)
.append(", username=").append(this.username)
.append(", pass=").append(this.password)
.append(", attemptCount=").append(this.attemptCount)
.append(", attemptTimeout=").append(this.attemptTimeout)
.append(", attemptDelay=").append(this.attemptDelay).toString());
}
Tibjms.setExceptionOnFTSwitch(true);
TibjmsTopicConnectionFactory topicConnectionFactory = new TibjmsTopicConnectionFactory(providerUrl);
topicConnectionFactory.setUserName(username);
topicConnectionFactory.setUserPassword(password);
topicConnectionFactory.setConnAttemptCount(attemptCount);
topicConnectionFactory.setConnAttemptTimeout(attemptTimeout);
topicConnectionFactory.setConnAttemptDelay(attemptDelay);
topicConnectionFactory.setReconnAttemptCount(attemptCount);
topicConnectionFactory.setReconnAttemptTimeout(attemptTimeout);
topicConnectionFactory.setReconnAttemptDelay(attemptDelay);
CachingConnectionFactory cachingConnectionFactory = new CachingConnectionFactory(topicConnectionFactory);
cachingConnectionFactory.setReconnectOnException(true);
cachingConnectionFactory.setExceptionListener(new JMSExceptionListener());
cachingConnectionFactory.setSessionCacheSize(10);
TransactionAwareConnectionFactoryProxy transactionAwareConnectionFactoryProxy = new TransactionAwareConnectionFactoryProxy();
transactionAwareConnectionFactoryProxy.setTargetConnectionFactory(cachingConnectionFactory);
transactionAwareConnectionFactoryProxy.setSynchedLocalTransactionAllowed(true);
return transactionAwareConnectionFactoryProxy;
}
Thanks, Nick
