0

I'm trying to execute an application under (reasonable) load. What is happening under load is that when trying to place a message onto a queue, the application stalls for about 4 seconds before completing the send. The strange part is that immediately after doing this, the next message takes a matter of milliseconds to place onto the queue. The message is in fact the same message - so the message size isn't a factor.

The application is using Spring Boot 2.1.6, Apache Qpid 0.43.0 as the JMS/AMQP provider.

The message bus being used is Azure ServiceBus, but I have observed the same behaviour using Artemis.

On the Apache Qpid JmsConnectionFactory, I've tried fiddling with the properties "forceSyncSend".

I've tried using the Spring Boot CachingConnectionFactory to cache message producers only. I have increased the default cache size from 1 to 20 without any success.

I've looked at the JmsTemplate parameters but can't find any parameters in regard to message producers (plenty with listeners but that's another story).

The code doing the sending is quite simple:

private void sendToQueue(Object message, String queueName) {
        jmsTemplate.convertAndSend(queueName, message, (Message jmsMessage) -> {
            jmsMessage.setStringProperty(OBJECT_TYPE_PARAMETER, message.getClass().getSimpleName());
            return jmsMessage;
        });

Is there anything obvious to try? Are there any tuning parameters to stop this stalling happening?

The load on the system is not trivial, but it is not excessive (it needs to go a lot higher than where it is at the moment!)

Any ideas?

Vadim Kotov
  • 8,084
  • 8
  • 48
  • 62
Ian Rowlands
  • 153
  • 1
  • 11
  • Instead of trying things randomly, profile the application and find out the root cause. Maybe check the [issue tracker](https://issues.apache.org/jira/browse/QPID-4343) for possible information as well. – Kayaman Sep 26 '19 at 09:08
  • It is a little challenging to profile because the application is in an Azure Kubernetes cluster. I know from the logging that the stalling thread isn't doing much, and there aren't a massive number of active threads during that stall period. Hence I have no idea why it is stalling. – Ian Rowlands Sep 26 '19 at 12:44
  • do you seen anything like this in the logs? "Message send delayed by 5s due to broker enforced flow control" – codebrane Sep 26 '19 at 13:30
  • Nothing in the logs to indicate any errors. – Ian Rowlands Sep 26 '19 at 22:42
  • I'd also recommend you investigate the flow-control aspect. To be clear, flow-control is not an error. Its a fundamental part of the AMQP protocol and could very well be enforced silently. – Justin Bertram Sep 27 '19 at 17:28
  • Any ideas on how to display this in QPID? I've seen flags like "qpid.flow_control_wait_failure" and "qpid.flow_control_wait_notify_period" but they appear in the documentation for the AMQP 0.8/0.9 client - I'm using the 1.0 client. Or is there a simpler way to see these types of messages (assuming they exist). – Ian Rowlands Oct 01 '19 at 06:24

0 Answers0