1

I am using spring-rabbit-1.7.3.RELEASE.jar

I have defined a SimpleMessageListenerContainer in my xml with shutdownTimeout parameter.

bean id="aContainer"
class="org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer">
    <property name="connectionFactory" ref="rabbitConnectionFactory" />
    <property name="queueNames" value="aQueue" />
    <property name="adviceChain" ref="retryAdvice" />
    <property name="acknowledgeMode" value="AUTO" />
    <property name="shutdownTimeout" value="900000" />
</bean>

When my service shuts down and there are still messages in "aQueue", I expect that the shutdownTimeout would allow the messages to get processed. But this doesn't seem to happen.

On further investigation I found out that the await() method defined in SimpleMessageListenerContainer is always returning true.

this.cancellationLock.await(Long.valueOf(this.shutdownTimeout), TimeUnit.MILLISECONDS);

I would like to understand how the logic for await works, how it acquires lock and what additional configuration is required at my end to make the code work.

GGKamal
  • 21
  • 6

1 Answers1

0

It waits for the consumers on-the-fly, those who are busy to process already fetched but not yet acknowledged messages. Nobody is going to poll fresh messages from the queue during shutdown.

The ActiveObjectCounter awaits on the all internal CountDownLatchs to be released. And that happens when we:

public void handleShutdownSignal(String consumerTag, ShutdownSignalException sig) {

So, that really might be a fact that all your consumers (private volatile int concurrentConsumers = 1; by default) are cancelled and released during that shutdownTimeout.

But again: no body is going to poll new messages from the Broker when the state is shutdown.

Artem Bilan
  • 113,505
  • 11
  • 91
  • 118
  • If you want to wait until the queue is empty, you could wait until you get a `ListenerContainerIdleEvent` before stopping the container - see [Detecting Idle Asynchronous Consumers](https://docs.spring.io/spring-amqp//reference/html/_reference.html#idle-containers). – Gary Russell Jan 18 '18 at 20:38
  • @Artem Bilan The way I am testing this, by adding thread.sleep() of 5 inutes to the serviceActivator processing the message from the Queue. So my expectation is that SimpleMessageListenerContainer will wait for the existing message to get processed, and then shut down. But I don't see that happening. I don't expect new messages to be polled, just the current one to be processed in the defined shutdown timeout period. – GGKamal Jan 19 '18 at 18:18
  • @GaryRussell Don't want to wait till queue is empty, just wait for the current message to be processed and then service to shut down gracefully. – GGKamal Jan 19 '18 at 18:19
  • Any chances to have a simple Spring Boot project to let us to play from our side? Does your observation actually mean that the message on-the-fly isn't processed correctly there ? – Artem Bilan Jan 19 '18 at 18:21
  • 1
    @ArtemBilan Yes, working on a Simple Spring Boot project with basic set up to ensure that nothing else in configuration is interfering with the expected functionality. – GGKamal Jan 19 '18 at 18:50
  • @ArtemBilan 'shutdowntimeout' property does seem to work for a Simple Spring boot project. Is there a configuration that configuration that could potentially be interfering with the functionality. – GGKamal Jan 19 '18 at 23:53
  • Well, I think you have some thread shifting between listener container and the mentioned service-activator. – Artem Bilan Jan 20 '18 at 01:37
  • @ArtemBilan You are right. I noticed that when shutdown all Consumers are cancelled and released. I see this in the logs: Restarting Consumer@df3f8ec: tags=[{amq.ctag-V5SbphFJMpfgcrtlRTYyAg=aQueue}], channel=Cached Rabbit Channel: AMQChannel(amqp://guest@127.0.0.1:5672/,1), conn: Proxy@47a5ada6 Shared Rabbit Connection: null, acknowledgeMode=AUTO local queue size=0 How I can prevent the cancellation and release of consumers during shutdown or change the sequence? – GGKamal Jan 23 '18 at 19:05
  • Well, seems for me that is by design. When we done with the consumer to prevent any new messages to be pulled from the Broker we cancel it and log the callback message when receive an answer from the broker on the matter. How does it effect your application though? – Artem Bilan Jan 23 '18 at 19:11
  • @ArtemBilan We want to implement a graceful shutdown. If there is message in queue that is being processed, then the queue should wait for the shutdownTimeout interval to process it. Currently, when shutdown happens, the message which is getting processed gets lost. – GGKamal Jan 23 '18 at 19:19
  • I tried adding the shutdownTimeOut property to SimpleMessageListenerContainer, hoping that the queue will wait for this time period to process the current message before shutting down, but when the ActiveObjectCounter.await(Long timeout, TimeUnit timeUnit) method always returns true. Since if (this.locks.isEmpty()) is always true. – GGKamal Jan 23 '18 at 19:21
  • Well, it is going to wait only if you block the listener thread. If you shift the processing to different thread, you are on your own and should ensure some other way for a graceful shutdown – Artem Bilan Jan 23 '18 at 19:21
  • @GaryRussell as mentioned in the previous comment, I created a simple spring boot application, where the shutdownTimeout works just fine. However, in my actual application it doesn't. Upon further investigation on both the applications I found out that in the simple spring boot application the doShutdown() method of SimpleMessageListenerContainer gets called during shutdown. This method internally calls ActiveObjectCounter.await() and finds the necessary lock. – GGKamal Jan 23 '18 at 23:29
  • @ArtemBilan But in my application, during shutdown the flow goes to line if (!isActive(this.consumer) || aborted) of run() method in SimpleMessageListenerContainer. This returns false and then the flow proceeds to restart(this.consumer). The restart method releases the acquired lock. After this the doShutdown() method of SimpleMessageListenerContainer gets called, but the lock has already been released and there by the application just shuts down instead of processing the current message. What should I do to correct the behavior of my application? – GGKamal Jan 23 '18 at 23:32
  • What you mean “but”? Looks like you report different behavior for the same cause. Maybe you use there different versions of the Spring AMQP? I mean in your real app and that app for testing. What version is there though anyway, please? – Artem Bilan Jan 23 '18 at 23:35
  • @ArtemBilan Yes, both the applications are behaving differently. Version for the jar is same in both of them : spring-rabbit-1.7.3.RELEASE.jar – GGKamal Jan 23 '18 at 23:47
  • Ok. How about to try the latest one: https://github.com/spring-projects/spring-amqp/releases? At least `1.7.5`. Seems for me we had some fix recently on the matter... – Artem Bilan Jan 23 '18 at 23:55
  • Ok, let me give that a try. – GGKamal Jan 24 '18 at 00:00
  • Changing the version did not seem to help. – GGKamal Jan 24 '18 at 22:13
  • OK. No chances to help you without the simple project for us to reproduce from our side. Thanks for understanding. – Artem Bilan Jan 24 '18 at 22:14