1

Summary - Message loss during systematic shutdown of application.

I have a application written in spring integration and I am consuming requests from external systems using 'jms:message-driven-channel-adapter'. This is the configuration of the channel adapter - 

<jms:message-driven-channel-adapter id="InboundAdapter" destination="InQueue"
                                        connection-factory="connectionFactory"
                                        channel="responseChannel"
                                        error-channel="errorChannel"
                                        acknowledge="transacted"
                                        receive-timeout="50000"
                                        auto-startup="true"/>

my response channel looks like this - 

 <int:chain id="processorChain" input-channel="responseChannel">
        <int:service-activator method="doProcess" ref="inputProcessor" />

        <int:router id="inputRouter" method="route">
            <int:mapping value="REQUIRED" channel="builderChannel"/>
            <int:mapping value="NOT_REQUIRED" channel="TerminateChannel"/>
            <bean id="Router" class="xxx.xxx.Router">
            </bean>
        </int:router>
    </int:chain>

Now when i perform 'kill -9 pid', then I see that the message is rolled back to the queue and it is all good. But when I am performing 'kill pid', than the message is lost somewhere. I have enabled teh jms logs and I can see that JMS listener is waiting for the message in transit to complete before closing the JMS consumer but still I don't the message rolling back to my queue. This is the logs snippet that I see in my logs

trace: 15.11.18 15:42:46.619 [Thread-10] DEBUG org.springframework.jms.listener.DefaultMessageListenerContainer - Waiting for shutdown of message listener invokers trace: 15.11.18 15:42:46.619 [Thread-10] DEBUG org.springframework.jms.listener.DefaultMessageListenerContainer - Still waiting for shutdown of 1 message listener invokers (iteration 0)

After this logs, it is calling the service activators which are defined in the response channel.

Can someone please shed some light on above behavior, I was expecting that when we issue kill command than all messages which are in transit should be rolled back to the queue, instead of this, it is trying to call the activators defined with in channel and after calling last component which is router, it just finishes.

Any help on this topic will be really helpful!!

After some more debugging and banging my head with the system I found the exact scenario where it is happening. Let me put it thru to see if it makes sense. When a systematic shutdown is triggered, JMS is waiting for the messages in transit to finish before stopping the application, till this point everything is good. Currently this chain is being executed - 

<int:chain input-channel="inputchain">
    <int:transformer id="xxx" method="transform">
        <bean class="xxx" />
    </int:transformer>

    <int:service-activator  id="xxx" method="doProcess">
        <bean class="xx">
            <constructor-arg ref="xxx"/>
        </bean>
    </int:service-activator>

    <int:service-activator id="xxx" ref="rulesProcessor" method="doProcess"/>
    <int:service-activator id="xxx" ref="xxx" method="doProcess"/>

    <!-- Existing Flow Continues -->
    <int:router id="xxxRequiredRouter" method="xxxRequired">
        <int:mapping value="Required" channel="firstChannel"/>
        <int:mapping value="NotRequired" channel="secondChannel"/>
        <bean id="xxxRouter" class="xxx.Router" />
        </bean>
    </int:router>

</int:chain>

<int:chain input-channel="secondChannel">

   some logic

</int:chain>

So the thread finishes this chain and exits gracefully. It doesn't call my next chain 'secondChannel'. My transaction boundary doesn't finish on this chain 'inputchain', it finishes at the end of secondChannel. So I have not comnmitted this transction to database as transaction boundary is at the end of next chain, so this is not available in DB and application is thinking that chain execution is finished so it is complete, so it is not rolled back to teh queue as well. So in teh end I don't have this message in my database and it is not in queue.

So is this the case that only the chain which is being executed when shutdown was triggered will finish and it will not delegate processing to subsequent chains?

Vaibs
  • 51
  • 5
  • Why do you say a "message loss", if you explain in the end that everything goes well until the router? I would say this is fully normal behavior when we call stop, but let in-flight tasks to be finished. – Artem Bilan Nov 15 '18 at 16:48
  • Hi Artem, I am calling it message loss because after calling the router, it should have gone to the next chain (builderChannel). But after router execution there is nothing else happening. Since execution didn't finished so my transaction is not committed to database and since the message didn't went back to queue so I can not process it when server comes back again. So effectively I have lost the message and I will have to ask the source systems to play it again manually which is not a acceptable solution. – Vaibs Nov 16 '18 at 09:27
  • If your all process is in the same thread, then I would suggest to use a `JmsTransactionManager` for that ``. Also since you mentioned data based it might even better to wrap both transaction managers into the `ChainedTransactionManager`. See here for more info: https://www.javaworld.com/article/2077963/open-source-tools/distributed-transactions-in-spring--with-and-without-xa.html – Artem Bilan Nov 16 '18 at 17:17

1 Answers1

0

A SIGTERM kill will wait for the thread to complete its work.

Eventually the container will interrupt it, but that will only help if it's doing something that's interruptible.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • Yes, That was teh expectation that it should either finish the task whicc is in progress and if it is not committed than the message should be rolled back to the queue. But I don't see any of this is happening, my transaction is not complete so it is not committed and the message didn't went back to the queue. either – Vaibs Nov 16 '18 at 09:31
  • That's not possible unless the thread is stuck in some uninterruptible code and never returns to the container and the JVM never shuts down. – Gary Russell Nov 16 '18 at 15:04
  • Thanks for reply Gary. I know this is strange and it should have worked the way we are expecting but somehow I am not getting the desired results. I was thinking if I should use client acknowledgement rather than using transacted. Do you think that will help me as the control will be in client code rather than relying on framework. Do you think it would be a good approach? – Vaibs Nov 20 '18 at 09:10
  • It should not be necessary; if you can post a small project somewhere, that reproduces the behavior, I will take a look at it. – Gary Russell Nov 20 '18 at 13:58
  • After some more debugging and banging my head with the system I found the exact scenario where it is happening. I have updated the actual issue with the findings. – Vaibs Nov 21 '18 at 16:36
  • I am having trouble parsing your explanation "as transaction boundary is at the end of next chain," There is no reason we won't route to the second chain, unless `secondChannel` is a queue or executor channel, which would not make sense since it won't participate in the transaction then. As I said, post a complete example that exhibits the behavior and I will take a look; I can't "guess" what the rest of your flow looks like. – Gary Russell Nov 21 '18 at 16:48
  • Thanks Gary, I am working on a sample application that I can send across. In the meantime can you tell me if you have idea on this point-when shutdown is triggered,I can see this message in logs- EventDrivenConsumer - Removing {chain:secondChain} as a subscriber to the 'secondChannel' channel stopped secondChain and MessageDeliveryException: Dispatcher has no subscribers for channel 'secondChannel'.This is happening before JMS connections are closed,so I am assuming that this is the reason the message has not routed to this channel(secondChannel) as it has been already stopped. – Vaibs Nov 22 '18 at 15:15
  • Right; that makes sense - the application context is closed so the endpoints are unsubscribed. The dispatcher has no subscribers exception will then cause the transaction to rollback and put the message back on the queue. So, again, no message loss should occur. – Gary Russell Nov 22 '18 at 15:24