Mule ESB: why AMQP is shutting down in-between the flow with error message (Received shutdown signal for consumer tag)

Question

I'm using AMQP connector,and my queues are present in cluster envt. I'm not sure why this message is populated always. But this is not affecting any success running flow. But it is trying to try connections always.

I'm using Amqp transport jar: 3.4.4 ( Even I tried importing with latest jar 3.4.6 jar) same error has happening again. Amqp client: 3.2.1. I'm totally confused. Because I tried the same queues which is not present in cluster it works fine. Please suggest me where i'm doing wrong.

Using Ack( Manual) changed to 'MULE_AUTO' also. Helpless.

      WARN  2015-01-29 14:47:29,639 [amqpReceiver.01]    org.mule.transport.amqp.internal.endpoint.receiver.MessageReceiverConsumer:    Received shutdown signal for consumer tag: amq.ctag-13inp2uTSjUnIgiNjyFYAA, the   message receiver will try to restart.

 com.rabbitmq.client.ShutdownSignalException: connection error; reason: java.io.EOFException
at com.rabbitmq.client.impl.AMQConnection.startShutdown(AMQConnection.java:715)
at com.rabbitmq.client.impl.AMQConnection.shutdown(AMQConnection.java:705)
atcom.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:564)
 Caused by: java.io.EOFException
at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290)
at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95)
at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:131)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:533)
 WARN  2015-01-29 14:47:29,642 [amqpReceiver.03]     org.mule.transport.amqp.internal.endpoint.receiver.MessageReceiverConsumer: Received shutdown signal for consumer tag: amq.ctag-G-xCK-Uj-0aF7LcrYIuo3A, the message receiver will try to restart.
com.rabbitmq.client.ShutdownSignalException: connection error; reason: java.io.EOFException
at   com.rabbitmq.client.impl.AMQConnection.startShutdown(AMQConnection.java:715)
 at com.rabbitmq.client.impl.AMQConnection.shutdown(AMQConnection.java:705)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:564)
  Caused by: java.io.EOFException
at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290)
at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95)
at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:131)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:533)
  WARN  2015-01-29 14:47:29,642 [amqpReceiver.02]    org.mule.transport.amqp.internal.endpoint.receiver.MessageReceiverConsumer:   Received shutdown signal for consumer tag: amq.ctag-seeeDsnuB_1sSuo97C8OdQ, the   message receiver will try to restart.
com.rabbitmq.client.ShutdownSignalException: connection error; reason: java.io.EOFException
at com.rabbitmq.client.impl.AMQConnection.startShutdown(AMQConnection.java:715)
at com.rabbitmq.client.impl.AMQConnection.shutdown(AMQConnection.java:705)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:564)
  Caused by: java.io.EOFException
 at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290)
at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95)
at  com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:131)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:533)
   ERROR 2015-01-29 14:47:29,665 [AMQP Connection]   org.mule.exception.DefaultSystemExceptionStrategy: 
   ********************************************************************************
  Message               : Connection shutdown detected for: AMQP_Connector
  Code                  : MULE_ERROR--2
  --------------------------------------------------------------------------------
  Exception stack is:
   1. null (java.io.EOFException)
    java.io.DataInputStream:290 (null)
     2. connection error; reason: java.io.EOFException     (com.rabbitmq.client.ShutdownSignalException)
    com.rabbitmq.client.impl.AMQConnection:715 (null)
  3. Connection shutdown detected for: AMQP_Connector  (org.mule.transport.ConnectException)
   org.mule.transport.amqp.internal.connector.AmqpConnector$1:296   (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/transport/ConnectEx    ception.html)
      --------------------------------------------------------------------------------
      Root Exception stack trace:
      java.io.EOFException
at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290)
at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95)
at     com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:131)
+ 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for    everything)
         ********************************************************************************
   INFO  2015-01-29 14:47:29,696 [AMQP Connection ]  org.mule.exception.DefaultSystemExceptionStrategy: Exception caught is a   ConnectException, attempting to reconnect...
ERROR 2015-01-29 14:47:29,698 [amqpReceiver.03]    org.mule.exception.DefaultSystemExceptionStrategy: 
 ********************************************************************************
    Message               : Impossible to create new channels on     connection: amqp://test@11.11.2000.111:1234/
    Code                  : MULE_ERROR--2
        --------------------------------------------------------------------------------
Exception stack is:
  1. clean connection shutdown; reason: Attempt to use closed connection   (com.rabbitmq.client.AlreadyClosedException)
com.rabbitmq.client.impl.AMQConnection:168 (null)
 2. Impossible to create new channels on connection:        amqp://test@11.11.2000.111:1234/ (org.mule.transport.ConnectException)
    org.mule.transport.amqp.internal.client.ChannelHandler:204     (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/transport/ConnectEx   ception.html)

  ERROR 2015-01-29 15:12:51,604 [amqpReceiver.34] org.mule.transport.amqp.internal.endpoint.receiver.MultiChannelMessageSubReceiver: Failed to restart: MultiChannelMessageSubReceiver(.....)

Edit:

       org.mule.api.MessagingException: Failed to ack message w/deliveryTag: 1 on channel: AMQChannel(amqp://ezibuy@10.10.19.159:5672/,1) (org.mule.api.DefaultMuleException). Message payload is of type: String
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:32)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:58)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.chain.DefaultMessageProcessorChain.doProcess(DefaultMessageProcessorChain.java:94)
at org.mule.processor.chain.AbstractMessageProcessorChain.process(AbstractMessageProcessorChain.java:67)
at org.mule.processor.chain.InterceptingChainLifecycleWrapper.doProcess(InterceptingChainLifecycleWrapper.java:50)
at org.mule.processor.chain.AbstractMessageProcessorChain.process(AbstractMessageProcessorChain.java:67)
at org.mule.processor.chain.InterceptingChainLifecycleWrapper.access$001(InterceptingChainLifecycleWrapper.java:22)
at org.mule.processor.chain.InterceptingChainLifecycleWrapper$1.process(InterceptingChainLifecycleWrapper.java:66)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:24)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:58)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.chain.InterceptingChainLifecycleWrapper.process(InterceptingChainLifecycleWrapper.java:61)
at org.mule.exception.TemplateMessagingExceptionStrategy.route(TemplateMessagingExceptionStrategy.java:139)
at org.mule.exception.TemplateMessagingExceptionStrategy.handleException(TemplateMessagingExceptionStrategy.java:45)
at org.mule.execution.HandleExceptionInterceptor.execute(HandleExceptionInterceptor.java:37)
at org.mule.execution.HandleExceptionInterceptor.execute(HandleExceptionInterceptor.java:14)
at org.mule.execution.BeginAndResolveTransactionInterceptor.execute(BeginAndResolveTransactionInterceptor.java:54)
at org.mule.execution.ResolvePreviousTransactionInterceptor.execute(ResolvePreviousTransactionInterceptor.java:44)
at org.mule.execution.SuspendXaTransactionInterceptor.execute(SuspendXaTransactionInterceptor.java:50)
at org.mule.execution.ValidateTransactionalStateInterceptor.execute(ValidateTransactionalStateInterceptor.java:40)
at org.mule.execution.IsolateCurrentTransactionInterceptor.execute(IsolateCurrentTransactionInterceptor.java:41)
at org.mule.execution.ExternalTransactionInterceptor.execute(ExternalTransactionInterceptor.java:48)
at org.mule.execution.RethrowExceptionInterceptor.execute(RethrowExceptionInterceptor.java:28)
at org.mule.execution.RethrowExceptionInterceptor.execute(RethrowExceptionInterceptor.java:13)
at org.mule.execution.TransactionalErrorHandlingExecutionTemplate.execute(TransactionalErrorHandlingExecutionTemplate.java:109)
at org.mule.execution.TransactionalErrorHandlingExecutionTemplate.execute(TransactionalErrorHandlingExecutionTemplate.java:30)
at org.mule.processor.AsyncInterceptingMessageProcessor$AsyncMessageProcessorWorker.doRun(AsyncInterceptingMessageProcessor.java:181)
at org.mule.work.AbstractMuleEventWork.run(AbstractMuleEventWork.java:39)
at org.mule.work.WorkerContext.run(WorkerContext.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: org.mule.api.DefaultMuleException: Failed to ack message w/deliveryTag: 1 on channel: AMQChannel(amqp://ezibuy@10.10.19.159:5672/,1)
at org.mule.transport.amqp.internal.processor.Acknowledger.ack(Acknowledger.java:63)
at org.mule.transport.amqp.internal.processor.Acknowledger.ack(Acknowledger.java:49)
at org.mule.transport.amqp.internal.processor.Acknowledger.process(Acknowledger.java:38)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:24)
... 32 more
Caused by: com.rabbitmq.client.AlreadyClosedException: channel is already closed due to channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - unknown delivery tag 1, class-id=60, method-id=80)
at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190)
at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:291)
at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:285)
at com.rabbitmq.client.impl.ChannelN.basicAck(ChannelN.java:1012)
at org.mule.transport.amqp.internal.processor.Acknowledger.ack(Acknowledger.java:59)

score 2 · Answer 1 · answered May 07 '15 at 04:04

There should be a delivery-tag flow variable that is used to keep track of the connection. This variable is required to be able to acknowledge the message. I suspect the flow variable is disappearing at some point. This can happen if the ACK is done in a separate thread from the one where the flow starts. The reason is that the delivery-tag variable can't be serialized/deserialized as it is moved to another thread.

Is anything being done asynchronously?

score 1 · Answer 2 · answered Jan 29 '15 at 23:27

1

You are probably bitten a bug present when using request-response or request-reply.

Please try to use 3.6 where the bug is fixed.

answered Jan 29 '15 at 23:27

Víctor Romero

5,107
2
22
32

Thanks for the response. Even after using the recent jar. I'm getting the new exception. Is there any possible option in AMQP in Mule, Not to stop the Channel connection, until transaction finished for each request?. Looking for your thoughts. – star Feb 06 '15 at 20:24
I'm getting the new exception even after adding jar,(Edited the exception details again)Please have a glance.Thanks. – star Feb 06 '15 at 20:30

Mule ESB: why AMQP is shutting down in-between the flow with error message (Received shutdown signal for consumer tag)

2 Answers2