JMS DefaultMessageListenerContainer not able to shutdown on Spring application context.close()

Question

I am listening to the Solace dynamic topic using spring integration as below:

<int-jms:message-driven-channel-adapter
    id="myJmsAdapter" channel="receiveChannel" container="jmsContainer" />

<bean id="jmsContainer"
    class="org.springframework.jms.listener.DefaultMessageListenerContainer">
    <property name="connectionFactory" ref="solaceConnectionFactory" />
      <property name="destinationName" value="${my.topic}/#{main.getCar_type_value()}" />
    <property name="pubSubDomain" value="true" />        
    <property name="sessionTransacted" value="false" />
    
</bean>

I am able to listen to the topic, consume the message, and process it. But on System.exit(0), even though the spring application context is closed inside the ShutdownHook, the JMS listener thread is hanging and not able to gracefully shutdown:

ctxt.registerShutdownHook();
...
  trigger = new ShutdownTrigger(maxWaitForResponseMillis, ctxt);
  trigger.startShutdownTimer();

    Runtime.getRuntime().addShutdownHook(new Thread() {
        public void run() {
            log.info("Exiting Main Thread!");
            ctxt.close();
        }
    });

The application shuts down properly on timeout, but when I try to exit before the timer using System.exit(0) form other class, it doesn't shutdown and the thread is hanging.

The timer function is as below:

   public void startShutdownTimer() {
      timer.schedule(new TimerTask() {
        @Override
        public void run() {
            log.info("Timer max wait has elapsed - triggering graceful 
     shutdown");
            
            // perform some task here on timeout..
            }
            log.info("Returning with exit code: " + exitCode);
            
            System.exit(exitCode); //this one works and it properly shutsdown
        }
    }, maxWaitForResponseMillis);
}

I also tried adding Timer.cancel before closing the context on the main class as below and it didnt work:

     Runtime.getRuntime().addShutdownHook(new Thread() {
    public void run() {
    log.info("Cancelling current timer task..")
        trigger.getTimer().cancel(); //it didnt work and the thread still hangs
        log.info("Exiting Main Thread!");
        ctxt.close();
    }
});

I tried taking out the Timer function and all its references to see if its this timer thread that is causing the issue, but no.. the application hangs even though the timer task is deleted.

Below is the thread stack during it was hanging:

"Thread-3" #26 prio=5 os_prio=0 tid=0x00007fcf48024800 nid=0x60ad in Object.wait() [0x00007fd026ef1000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.springframework.jms.listener.DefaultMessageListenerContainer.doShutdown(DefaultMessageListenerContainer.java:568)
- locked <0x00000000ee0c9ec0> (a java.lang.Object)
at org.springframework.jms.listener.AbstractJmsListeningContainer.shutdown(AbstractJmsListeningContainer.java:237)

Please help to figure out the issue and the solution...

Post the entire thread dump someplace like a GitHub Gist or PasteBin etc. — Gary Russell, Jan 26 '21 at 20:29
Please find the thread dump in the below location: https://gist.github.com/vathappan/ff41131a696b7638124278c5fd15a405 — merin george, Jan 27 '21 at 17:20

score 0 · Accepted Answer · answered Jan 27 '21 at 19:52

0

You are calling System.exit() on the listener container thread

"jmsContainer-1" #24 prio=5 os_prio=0 tid=0x00007fdbd8e50800 nid=0xb77 in Object.wait() [0x00007fdb9f1bc000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000000f02cfb68> (a com.tdsecurities.fxts.vs.FxVsCashBalanceApp$1)
    at java.lang.Thread.join(Thread.java:1245)
    - locked <0x00000000f02cfb68> (a com.tdsecurities.fxts.vs.FxVsCashBalanceApp$1)
    at java.lang.Thread.join(Thread.java:1319)
    at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:106)
    at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
    at java.lang.Shutdown.runHooks(Shutdown.java:123)
    at java.lang.Shutdown.sequence(Shutdown.java:167)
    at java.lang.Shutdown.exit(Shutdown.java:212)
    - locked <0x00000000c04335c8> (a java.lang.Class for java.lang.Shutdown)
    at java.lang.Runtime.exit(Runtime.java:109)
    at java.lang.System.exit(System.java:971)

This is causing a deadlock in the Application context (Thread-3 and Thread-5) (shutting down the container).

You probably need to hand off the System.exit() call to another thread, but I'd need to know what version of Spring you are using to analyze further. Otherwise it's hard to correlate the line numbers in the dump.

answered Jan 27 '21 at 19:52

Gary Russell

166,535
14
146
179

Hi Gary, Thanks for your response. I am using Spring 4.2.4. I also want to add that when the application starts it also schedule a timer task for 1 hour and what I am trying to do it after the application process the JMS response successfully, I am trying to exit out without waiting for the timer. I tried adding Timer.cancel() wondering if thats the thread causing the issue, but it didnt help. I will edit the original post with the timer function too. Thanks – merin george Jan 28 '21 at 13:59
As I said; the problem is that you are calling `System.exit()` on the container thread which is, in turn, via the hook, trying to shut down the container. By the way, 4.2 is no longer supported; 4.3 went out of support at the end of last year too. You should upgrade to 5.3 as soon as possible. – Gary Russell Jan 28 '21 at 14:38
Now having the proper line numbers, I can indeed confirm that the shutdown hook (trying to close the context) is blocked waiting for the listener container thread to terminate, and the listener container thread is blocked waiting for the shutdown hook to terminate. Hence an irrecoverable deadlock. – Gary Russell Jan 28 '21 at 20:27
Thanks Gary. I tried to handle exit out of the JMS chain, still its hanging. I have added the flow here: https://gist.github.com/vathappan/8b17e8dc5cc854319b072fc5b8686bbd Please take a look and help – merin george Jan 29 '21 at 15:19
That won't make any difference; the service is still processed on the same thread - you need to hand off the `System.exit()` call to another thread. – Gary Russell Jan 29 '21 at 15:52
Hi Gary, thanks for looking into it. Rather than a timer thread, how can I hand off the System.exit() to a separate thread right after processing the JMS response? – merin george Feb 02 '21 at 17:41
Add a `TaskExecutor` bean; e.g. a `SimpleAsyncTaskExecutor`; then `exec.execute(() -> System.exit(exitCode));` – Gary Russell Feb 02 '21 at 17:47

JMS DefaultMessageListenerContainer not able to shutdown on Spring application context.close()

1 Answers1