I have searched a few on the internet, but have not found anything for me hence posting here.
The error message "#method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - delivery acknowledgement on channel 8 timed out" is well describing the consumer_timeout (ack timeout) happening. I'm using CachingConnectionFactory.
As per the monitor internal, it keeps looping and trying to create a new channel but the channel number is the same as the previous one and again shutting down the same channel. It's not able to restart the consumer. I want to drop that old channel and create a new channel like before. Why its opening the same channel (i.e. channel 8) again after closing? Any suggestion would be a great help.
Below is the sample of the log for ref :
2022-08-09 09:25:27.688 INFO 1 --- [enerContainer-1] m.m.r.CartMessageListenerContainer : SimpleConsumer [queue=MYCART.JOBS, consumerTag=amq.ctag-SsW-lSN8lqAg3PUGJr5HDQp identity=2e829d86t] started
2022-08-09 21:27:01.124 ERROR 1 --- [.232.164.84:443] m.m.r.a.client.AmqpConnectionFactory$1 : Shutdown Signal: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - delivery acknowledgement on channel 8 timed out. Timeout value used: 432000 ms. This timeout value can be configured, see consumers doc guide to learn more, class-id=0, method-id=0)
2022-08-09 21:27:01.124 DEBUG 1 --- [.232.164.84:443] m.m.r.amqp.client.AmqpChannelListener : AMQP channel closed [false] channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - delivery acknowledgement on channel 8 timed out. Timeout value used: 432000 ms. This timeout value can be configured, see consumers doc guide to learn more, class-id=0, method-id=0)
2022-08-09 21:27:02.641 ERROR 1 --- [enerContainer-1] m.m.r.CartMessageListenerContainer : Consumer canceled - channel closed SimpleConsumer [queue=MYCART.JOBS, consumerTag=amq.ctag-SsW-lSN8lqAg3PUGJr5HDQp identity=2e829d86t]
2022-08-09 21:27:02.641 DEBUG 1 --- [enerContainer-1] m.m.r.a.client.AmqpConnectionFactory$1 : Closing cached Channel: PublisherCallbackChannelImpl: AMQChannel(amqp://xxxxxxxxxxxxxxxxx/,8)
2022-08-09 21:27:07.641 DEBUG 1 --- [enerContainer-1] m.m.r.CartMessageListenerContainer : Attempting to restart consumer SimpleConsumer [queue=MYCART.JOBS, consumerTag=amq.ctag-SsW-lSN8lqAg3PUGJr5HDQp identity=2e829d86t]
2022-08-09 21:27:07.658 DEBUG 1 --- [enerContainer-1] m.m.r.a.client.AmqpConnectionFactory$1 : Creating cached Rabbit Channel from PublisherCallbackChannelImpl: AMQChannel(amqp://xxxxxxxxxxxx:443/,8)
2022-08-09 21:27:07.658 DEBUG 1 --- [enerContainer-1] m.m.r.amqp.client.AmqpChannelListener : AMQP channel opened [false] 8
2022-08-09 21:27:07.685 INFO 1 --- [enerContainer-1] m.m.r.CartMessageListenerContainer : SimpleConsumer [queue=MYCART.JOBS, consumerTag=amq.ctag-gGFtDxSYhalJ0gWish3voQt identity=3a1d1b7bt] started
2022-08-10 09:27:55.005 ERROR 1 --- [.232.164.84:443] m.m.r.a.client.AmqpConnectionFactory$1 : Shutdown Signal: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - delivery acknowledgement on channel 8 timed out. Timeout value used: 432000 ms. This timeout value can be configured, see consumers doc guide to learn more, class-id=0, method-id=0)
2022-08-10 09:27:55.005 DEBUG 1 --- [.232.164.84:443] m.m.r.amqp.client.AmqpChannelListener : AMQP channel closed [false] channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - delivery acknowledgement on channel 8 timed out. Timeout value used: 432000 ms. This timeout value can be configured, see consumers doc guide to learn more, class-id=0, method-id=0)
2022-08-10 09:27:57.641 ERROR 1 --- [enerContainer-1] m.m.r.CartMessageListenerContainer : Consumer canceled - channel closed SimpleConsumer [queue=MYCART.JOBS, consumerTag=amq.ctag-gGFtDxSYhalJ0gWish3voQt identity=3a1d1b7bt]
2022-08-10 09:27:57.641 DEBUG 1 --- [enerContainer-1] m.m.r.a.client.AmqpConnectionFactory$1 : Closing cached Channel: PublisherCallbackChannelImpl: AMQChannel(amqp://xxxxxxxxxxxxxxxxxx:443/,8)
2022-08-10 09:27:57.641 DEBUG 1 --- [enerContainer-1] m.m.r.CartMessageListenerContainer : Attempting to restart consumer SimpleConsumer [queue=MYCART.JOBS, consumerTag=amq.ctag-gGFtDxSYhalJ0gWish3voQt identity=3a1d1b7bt]
2022-08-10 09:27:57.659 DEBUG 1 --- [enerContainer-1] m.m.r.a.client.AmqpConnectionFactory$1 : Creating cached Rabbit Channel from PublisherCallbackChannelImpl: AMQChannel(amqp://xxxxxxxxxxxxx:443/,8)
2022-08-10 09:27:57.659 DEBUG 1 --- [enerContainer-1] m.m.r.amqp.client.AmqpChannelListener : AMQP channel opened [false] 8
2022-08-10 09:27:57.687 INFO 1 --- [enerContainer-1] m.m.r.CartMessageListenerContainer : SimpleConsumer [queue=MYCART.JOBS, consumerTag=amq.ctag-RKgc4e0o1wii-OBE9ai69A identity=68a0033a] started
2022-08-10 21:28:01.145 ERROR 1 --- [.232.164.84:443] m.m.r.a.client.AmqpConnectionFactory$1 : Shutdown Signal: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - delivery acknowledgement on channel 8 timed out. Timeout value used: 432000 ms. This timeout value can be configured, see consumers doc guide to learn more, class-id=0, method-id=0)
AmqpConnectionFactory.java
@Slf4j
@Configuration
public class AmqpConnectionFactory implements InitializingBean, ApplicationListener<ContextClosedEvent> {
private static final Map<Class<? extends Throwable>,Boolean> EXCEPTION_MAP = new HashMap<>();
private static final long INITIAL_RETRY_INTERVAL = 1000L;
private static final int MAX_RETRY_ATTEMPTS = 20;
@Autowired
ApplicationContext applicationContext;
@Autowired
QueueConfig queueConfig;
private RabbitTemplate rabbitTemplate;
private RabbitAdmin rabbitAdmin;
private volatile CachingConnectionFactory cachingConnectionFactory;
private boolean initialized = false;
@Override
public void afterPropertiesSet() {
try {
initialize();
} catch (Exception ex) {
//For now, we're going to say this is ok
}
}
public boolean isInitialized() {
return initialized;
}
public void reinitialize(String host, String port, boolean useSsl, String username, String password) throws Exception {
queueConfig.setHost(host);
queueConfig.setPort(Integer.valueOf(port));
queueConfig.setSslEnabled(useSsl);
queueConfig.setUsername(username);
queueConfig.setPassword(password);
initialize();
}
public void initialize() throws Exception {
if (!StringUtils.hasLength(queueConfig.getHost()) || !StringUtils.hasLength(queueConfig.getUsername())) {
log.info("AMQP configuration is disabled or not complete");
return;
}
log.info("Attempting to connect: {}:{} as {} [ssl={}]", queueConfig.getHost(), queueConfig.getPort(), queueConfig.getUsername(), queueConfig.isSslEnabled());
if (queueConfig.isSslEnabled()) {
cachingConnectionFactory = new CachingConnectionFactory(
new SSLConnectionFactory(queueConfig)) {
@Override
public void onApplicationEvent(ContextClosedEvent event) {
waitForEngineTaskExecutor();
super.onApplicationEvent(event);
}
};
} else {
cachingConnectionFactory = new CachingConnectionFactory(
new BaseConnectionFactory(queueConfig)) {
@Override
public void onApplicationEvent(ContextClosedEvent event) {
waitForEngineTaskExecutor();
super.onApplicationEvent(event);
}
};
}
cachingConnectionFactory.addChannelListener(new AmqpChannelListener());
cachingConnectionFactory.addConnectionListener(new AmqpConnectionListener());
cachingConnectionFactory.setChannelCacheSize(queueConfig.getChannelCacheSize());
cachingConnectionFactory.setConnectionCacheSize(queueConfig.getConnectionCacheSize());
cachingConnectionFactory.setConnectionLimit(queueConfig.getConnectionLimit());
cachingConnectionFactory.setConnectionNameStrategy(f -> HostInfo.getHostname());
cachingConnectionFactory.setPublisherConfirmType(CachingConnectionFactory.ConfirmType.CORRELATED);
log.info("Connection Last cachingConnectionFactory {} ", cachingConnectionFactory );
try {
rabbitTemplate = new RabbitTemplate(cachingConnectionFactory);
rabbitTemplate.setRetryTemplate(amqpRetryTemplate());
} catch (Exception ex) {
log.warn("AMQP credentials may be insufficient for pub/sub capabilities");
}
try {
rabbitAdmin = new RabbitAdmin(cachingConnectionFactory);
rabbitAdmin.setRetryTemplate(amqpRetryTemplate());
rabbitAdmin.setIgnoreDeclarationExceptions(true);
} catch (Exception ex) {
log.warn("AMQP credentials may be insufficient for Admin capabilities");
}
initialized = true;
}
private void waitForEngineTaskExecutor() {
try {
ThreadPoolTaskExecutor engineTaskExecutor = (ThreadPoolTaskExecutor)applicationContext.getBean("engineTaskExecutor");
if (engineTaskExecutor != null) {
try {
while(engineTaskExecutor.getActiveCount() > 0) {
log.warn("There are {} jobs currently executing", engineTaskExecutor.getActiveCount());
Thread.currentThread().sleep(10000);
}
log.info("All jobs have been completed");
} catch (Exception ex) {
log.warn("Termination signal received: {}", ex.getMessage());
}
} else {
log.info("EngineTaskExecutor was not initialized at shutdown");
}
} catch (NoSuchBeanDefinitionException ex) {
//We don't have to wait b/c there is no engineTaskExecutor bean
}
}
@Retryable(include = {AmqpAuthenticationException.class, AmqpConnectException.class},
maxAttempts = MAX_RETRY_ATTEMPTS,
backoff = @Backoff(delay = INITIAL_RETRY_INTERVAL,
multiplier = ExponentialBackOffPolicy.DEFAULT_MULTIPLIER,
maxDelay = ExponentialBackOffPolicy.DEFAULT_MAX_INTERVAL))
public void waitForConnection() throws Exception {
connect();
}
@Retryable(include = {AmqpAuthenticationException.class, AmqpConnectException.class},
maxAttempts = Integer.MAX_VALUE,
backoff = @Backoff(delay = 5000L,
multiplier = ExponentialBackOffPolicy.DEFAULT_MULTIPLIER,
maxDelay = ExponentialBackOffPolicy.DEFAULT_MAX_INTERVAL))
public void waitForConnectionIndefinitely() throws Exception {
connect();
}
public boolean isConnected() {
try {
connect();
return true;
} catch (Exception ex) {
}
return false;
}
private void connect() throws Exception {
if (!initialized) initialize();
if (cachingConnectionFactory == null) throw new AmqpConnectException(new RuntimeException("Connection not yet intialized"));
try {
cachingConnectionFactory.createConnection();
} catch (AmqpAuthenticationException | AmqpConnectException ex) {
log.debug("AMQP connection failure: " + ex.getMessage());
throw ex;
}
}
@Bean
public RetryTemplate amqpRetryTemplate() {
//Note this retryTemplate may not handle initial connection failures
RetryTemplate retryTemplate = new RetryTemplate();
EXCEPTION_MAP.put(AmqpConnectException.class, true);
retryTemplate.setRetryPolicy(new SimpleRetryPolicy(MAX_RETRY_ATTEMPTS, EXCEPTION_MAP, true));
ExponentialBackOffPolicy backOffPolicy = new ExponentialRandomBackOffPolicy();
backOffPolicy.setInitialInterval(INITIAL_RETRY_INTERVAL);
backOffPolicy.setMultiplier(ExponentialBackOffPolicy.DEFAULT_MULTIPLIER);
backOffPolicy.setMaxInterval(ExponentialBackOffPolicy.DEFAULT_MAX_INTERVAL);
retryTemplate.setBackOffPolicy(backOffPolicy);
return retryTemplate;
}
public RabbitTemplate getRabbitTemplate() {
try {
if (!initialized) initialize();
} catch (Exception ex) {
log.error("Failed to initialize RabbitTemplate", ex);
}
return rabbitTemplate;
}
// RabbitAdmin Bean is defined in AmqpRabbitAdmin
public RabbitAdmin getRabbitAdmin() {
try {
if (!initialized) initialize();
} catch (Exception ex) {
log.error("Failed to initialize RabbitAdmin", ex);
}
return rabbitAdmin;
}
public CachingConnectionFactory getConnectionFactory() {
try {
if (!initialized) initialize();
} catch (Exception ex) {
log.error("Failed to initialize CachingConnectionFactory", ex);
}
return cachingConnectionFactory;
}
@Override
public void onApplicationEvent(ContextClosedEvent e) {
waitForEngineTaskExecutor();
}
}
CartMessageListenerContainer.java
@Slf4j
@Service
public class CartMessageListenerContainer extends DirectMessageListenerContainer implements SmartLifecycle, ApplicationListener<ContextClosedEvent> {
AmqpConnectionFactory amqpConnectionFactory;
CartMessageListener CartMessageListener;
LocalWorkerAmqpConfig localWorkerAmqpConfig;
LocalWorkerConfig localWorkerConfig;
@Autowired
CartMessageListenerContainer(AmqpConnectionFactory amqpConnectionFactory, CartMessageListener CartMessageListener, LocalWorkerAmqpConfig localWorkerAmqpConfig, LocalWorkerConfig localWorkerConfig) throws Exception {
log.info("Initializing CartMessageListenerContainer...");
this.amqpConnectionFactory = amqpConnectionFactory;
this.CartMessageListener = CartMessageListener;
this.localWorkerAmqpConfig = localWorkerAmqpConfig;
this.localWorkerConfig = localWorkerConfig;
amqpConnectionFactory.waitForConnection();
this.setAcknowledgeMode(AcknowledgeMode.MANUAL);
this.setAutoDeclare(Boolean.FALSE);
this.setAutoStartup(Boolean.TRUE);
this.setConnectionFactory(amqpConnectionFactory.getConnectionFactory());
this.setIdleEventInterval(30000);
this.setMessageListener(CartMessageListener);
this.setMismatchedQueuesFatal(Boolean.FALSE);
this.setMissingQueuesFatal(Boolean.FALSE);
this.setPrefetchCount(localWorkerConfig.getConcurrency());
}
@Override
public void start() {
log.debug("Starting CartMessageListenerContainer...");
try {
List<Queue> cartQueues = new ArrayList<>();
cartQueues.add(localWorkerAmqpConfig.localDefaultQueue());
if (localWorkerConfig.getDestinationId() != null && !localWorkerConfig.getDestinationId().isEmpty()) {
try {
Long.valueOf(localWorkerConfig.getDestinationId());
Queue destinationQueue = new Queue(WorkerQueueNames.getJobsSubscribeToName(localWorkerConfig.getDestinationId()), true, false, false);
cartQueues.add(destinationQueue);
} catch (Exception ex) {
log.warn("DestinationId [{}] is invalid or you do not have access to it", localWorkerConfig.getDestinationId());
}
}
this.setQueues(cartQueues.toArray(new Queue[cartQueues.size()]));
this.lazyLoad();
super.start();
} catch (Exception ex) {
log.error("Failed to start CartMessageListenerContainer", ex);
}
}
@Override
public void stop() {
log.info("Stopping CartMessageListenerContainer...");
super.stop();
}
@Override
protected void handleListenerException(Throwable ex) {
//Overriding this b/c default impl logs a stacktrace after the container shuts down
if (isActive()) {
// Regular case: failed while active.
// Invoke ErrorHandler if available.
invokeErrorHandler(ex);
}
}
@Override
public void onApplicationEvent(ContextClosedEvent e) {
log.info("Destroying CartMessageListenerContainer...");
destroy();
}
}
BaseConnectionFactory.java
@Slf4j
public class BaseConnectionFactory extends ConnectionFactory {
protected BaseConnectionFactory(QueueConfig queueConfig) {
this.setHost(queueConfig.getHost());
this.setPort(queueConfig.getPort());
this.setUsername(queueConfig.getUsername());
this.setPassword(queueConfig.getPassword());
this.setRequestedHeartbeat(queueConfig.getConnectionTimeout());
//This ConnectionFactory will be wrapped in Spring CachingConnectionFactory,
//which prefers to use its own recovery mechanisms
this.setAutomaticRecoveryEnabled(false);
this.setExceptionHandler(new BaseConnectionExceptionHandler());
//shutdown consumers once the corresponding queue is deleted rather than notifying them
Map<String, Object> clientProperties = new HashMap<>();
clientProperties.put("product", ProductInfo.SHORT_NAME);
clientProperties.put("information", ProductInfo.LONG_NAME);
clientProperties.put("copyright", ProductInfo.COPYRIGHT);
clientProperties.put("version", ProductInfo.VERSION);
clientProperties.put("hostname", HostInfo.getHostname());
Map<String, Object> clientCapabilities = new HashMap<>();
clientCapabilities.put("exchange_exchange_bindings", Boolean.TRUE);
clientCapabilities.put("consumer_cancel_notify", Boolean.TRUE);
clientCapabilities.put("basic.nack", Boolean.TRUE);
clientCapabilities.put("publisher_confirms", Boolean.TRUE);
clientCapabilities.put("connection.blocked", Boolean.TRUE);
clientCapabilities.put("authentication_failure_close", Boolean.TRUE);
clientProperties.put("capabilities", clientCapabilities);
this.setClientProperties(clientProperties);
}
protected static class BaseConnectionExceptionHandler extends ForgivingExceptionHandler {
@Override
public void handleUnexpectedConnectionDriverException(Connection conn, Throwable exception) {
log.trace("AMQP connection driver failure: {}", exception.getMessage());
}
@Override
public void handleConnectionRecoveryException(Connection conn, Throwable exception) {
// ignore java.net.ConnectException as those are
// expected during recovery and will only produce noisy
// traces
if (exception instanceof ConnectException) {
// no-op
} else {
log.warn("AMQP connection recovery warning: {}", exception.getMessage());
}
}
@Override
public void handleChannelRecoveryException(Channel ch, Throwable exception) {
log.warn("AMQP channel recovery warning: {}", exception.getMessage());
}
}
}
