JBoss AS7 hangs after MDB lock

Question

I'm currently getting an intermittent exception in JBoss AS7 that, unfortunately, I cannot yet reproduce.

We are currently running two applications that are basically set as producer/consumer of JMS messages. We are using the default HornetQ configuration and a pool of 5 MDBs.

Both applications starts doing ok, sending and receiving messages as expected. But after a while, all MDBs get locked (they each get a message but do not finish processing) and JBoss hangs some time after that showing the following message every ten minutes:

[org.jboss.ejb3.invocation] (Thread-21081 (HornetQ-client-global-threads-1636833629)) JBAS014134: EJB Invocation failed on component MyMessageListener for method public abstract void javax.jms.MessageListener.onMessage(javax.jms.Message): javax.ejb.EJBException: JBAS014516: Failed to acquire a permit within 10 MINUTES

From JBoss code in jarvana, it seems as if this error is set if a semaphore cannot be acquired:

 /**
 * Get an instance without identity.
 * Can be used by finders,create-methods, and activation
 *
 * @return Context /w instance
 */
public T get() {
    try {
        boolean acquired = semaphore.tryAcquire(timeout, timeUnit);
        if (!acquired)
            throw new EJBException("Failed to acquire a permit within " + timeout + " " + timeUnit);
    } catch (InterruptedException e) {
        throw new EJBException("Acquire semaphore was interrupted");
    }
...

The thing is, why the MDBs get locked? Shouldn't they timeout and continue processing? I've set the timeout to 5 minutes in standalone.xml file, but they never, ever seem to timeout.

<session-bean>
<stateless>
   <bean-instance-pool-ref pool-name="slsb-strict-max-pool"/>
</stateless>
<stateful default-access-timeout="5000" cache-ref="simple"/>
<singleton default-access-timeout="5000"/>
</session-bean>

Does anyone know what could be happening?

I would also, very gladly, accept any suggestions on how to simulate the problem or on other ways to set MDBs timeout.

Any help would be appreciated.

Thanks for your time.

Edit:

So I was finally able to reproduce the problem simply by sending the MessageListener to sleep for longer than the instance-acquisition-timeout. Following is the test code:

<!-- standalone.xml -->
<strict-max-pool name="mdb-strict-max-pool" max-pool-size="5" instance-acquisition-timeout="30" instance-acquisition-timeout-unit="SECONDS"/>

MDB:

@MessageDriven(name = "TestMDB", activationConfig = {
    @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue"),
    @ActivationConfigProperty(propertyName = "destination", propertyValue = "queue/myQueue"),
    @ActivationConfigProperty(propertyName = "acknowledgeMode", propertyValue = "Auto-acknowledge")//,
    })
public class TestMDB implements MessageListener {

private final static Logger LOGGER = Logger.getLogger(TestMDB.class
        .toString());

@Resource 
private MessageDrivenContext ctx;

private final static int sleepingTime = MyProperties.SLEEP_MILLISECS; 

/**
 * @see MessageListener#onMessage(Message)
 */
public void onMessage(Message rcvMessage) {
    ObjectMessage msg = null;
    Future<String> future = null;
    MyResource res = null;

    try {
        if (rcvMessage instanceof ObjectMessage) {
            msg = (ObjectMessage) rcvMessage;
            res = (MyResource)msg.getObject();

            LOGGER.info("Received resource: " + res);
            Thread.sleep(sleepingTime);
            LOGGER.info("Released resource: " + res);
        } else {
            LOGGER.warning("Message of wrong type: "
                    + rcvMessage.getClass().getName());
        }
    } catch (JMSException e) {
        throw new RuntimeException(e);
    } catch (InterruptedException e) {
        throw new RuntimeException(e);
    }
}
}

Answer 1

It seems as if this behavior is to be expected and the issue is probably application related.

Below is a summary of the problem:

1. We have an MDB instance;
2. The MDB receives messages from queue Q1;
3. We send a message to queue Q1;
4. MDB tries to process it, but since we simulate a process that takes longer than the instance-acquisition-timeout, HornetQ timeouts and we get the exception: "Failed to acquire a permit within X MINUTES";
5. Unfortunately, the MDB continues processing and only is released after the sleep time
6. If we stop JBoss before the MDB release, the application continues its processing, since the messages are persistent and it knows we didn't finish the MDB#onMessage method;
7. But since we haven't identified the application problem, the lock happens again and we get an exception loop;

The ideal solution would be to discover why the MDBs get locked, but until then, since the application is already in production, we need a fall-back plan.

I believe one solution would be to interrupt the MDB thread after we reach Hornet's wait timeout (or on transaction Timeout - @ActivationConfigProperty(propertyName = "transactionTimeout", propertyValue = "30")), but there doesn't seem to be an easy configuration (yet) to do this in JBoss AS7.

The sad way would be to time the MDB thread execution and force an interruption within the onMessage method. This could easily be achieved with an ExecutorService (http://stackoverflow.com/questions/6460664/how-can-i-interrupt-method-execution-by-time)

For anyone experiencing the same problems, I found some similar discussions on the web:

http://bryanpendleton.blogspot.com.br/2009/05/timertask-exception-handling.html https://community.jboss.org/thread/162822

Hope this helps.

And hope I can detect whatever is wrong with the application.

Answer 2

I also encountered the issue, and we found the root cause finally by check the threads of JBoss by the following command.

jstack $pid

And we found a thread deadlock. after we fix the deadlock, JBoss didn't halt any more.