0

I have one server, on which I have one publisher which sends messages to a single queue. Reading from this queue I have 5 consumers, each on its own JVM. The idea is that the publisher's messages should be consumed as soon as possible by whichever consumer(s) is free. Sometimes all 5 will be free and then ActiveMQ presumably chooses one to receive/dequeue the message (?).

All messages are non-persistent. I'm using ActiveMQ pretty much out the box with just 1 queue and zero tinkering to any config files. I'm also not using transactions.

The publisher logs the time in millseconds right after it's returned:

public void sendMessage(String text) {
    TextMessage message = null;
    try {
        message = session.createTextMessage(text);
        producer.send(message);
        System.out.println("JUST FINISHED SENDING MESSAGE " + System.currentTimeMillis());
    } catch (JMSException e) {
        e.printStackTrace();
    }
}

Each consumer (running its own JVM) is listening on that queue for bursts of 2 minutes at time:

        Message message = consumer.receive(120000);
        if (message instanceof TextMessage) {
            TextMessage textMessage = (TextMessage) message;
            text = textMessage.getText();
            System.out.println("MESSAGE RECEIVED " + System.currentTimeMillis());
        }

Usually consumers will log "MESSAGE RECEIVED" at exactly the same time in milliseconds as the "JUST FINISHED SENDING MESSAGE", which is perfect. But sometimes there is an inexplicable delay of around 15 milliseconds between the publish and the consume, even when all consumers are sitting free. Given all processes are on the same server, and that latency is absolutely critical, I'm frustrated that there's sometimes this delay.

  1. Is this to be expected when running multiple consumers from 1 queue, or is that irrelevant?
  2. Is there anything I can do to mitigate the delay?

Thanks in advance for any useful advice.

S Clark
  • 33
  • 4
  • Even if all processes run on the same server, the communication still runs over the network interface. And honestly, the fact that that you have a dely of **only** 15 ms is - from my point of view - pretty amazing. – Turing85 Aug 15 '20 at 15:32
  • https://stackoverflow.com/questions/25121094/spring-jmsactivemq-delayed-delivery-of-messages – Shivang Agarwal Aug 15 '20 at 15:35
  • @ShivangAgarwal do you know if, after scheduled message support has been enabled, setting the delay explicitly to `0` will reduce the delay? – Turing85 Aug 15 '20 at 15:38
  • @ShivangAgarwal that article is someone looking for ways to explicitly delay messages on purpose. I'm trying to do the opposite. Are you actually suggesting scheduled messaging with a 0 delay as a means to eliminate occasional 15 millisecond delays, or did you post that in error? – S Clark Aug 15 '20 at 16:22
  • @Turing85 thanks for the message -- if this is as good as it gets then so be it. The fact it usually consumes messages within 1 millisecond led me to hope that such a latency was achievable all the time, and indeed i was keen to understand if having multiple consumers of the same queue (where only one consumer can win the race) would be expected to introduce occasional delays due to contention or whatever. – S Clark Aug 15 '20 at 16:23
  • 1
    You might try using [ActiveMQ Artemis](http://activemq.apache.org/components/artemis/) instead of ActiveMQ 5.x. Performance is generally better given the broker's non-blocking architecture. It was designed with performance and scalability as high priorities. – Justin Bertram Aug 15 '20 at 18:06
  • 1
    From my own experience, ActiveMQ can handle 0ms (or close to that) streams given the right setup. [Tim Bish's answer](https://stackoverflow.com/a/63428330/6580047) seems to get the right path to troubleshoot that, and a `tcpdump` would probably hint where's the slugish part for you to move forward this line of thinking. – Gustavo Kawamoto Aug 17 '20 at 13:27

1 Answers1

4

There are many reasons for short delays like this when dealing with network consumers and a central broker. The CPU scheduling of either the broker or the consumer along with work done managing the network interface could lead to small hiccups. You might also be seeing GC pauses as the JVM on the consumer or the broker cleans up which can delay dispatch or consumption. And of course inter-thread singaling can sometime have short delays depending on various things running alongside so these short pauses are really something I'd invest a ton of time in unless you get to the point where that is the last issue you need to solve out of all the other likely things you aren't dealing with yet.

Justin Bertram
  • 29,372
  • 4
  • 21
  • 43
Tim Bish
  • 17,475
  • 4
  • 32
  • 42