ActiveMQ Artemis performance degradation compared to "Classic"

Question

I'm working on a migration from ActiveMQ "Classic" 5.15.4 to ActiveMQ Artemis 2.17.0, and I observed a performance degradation. I tested with 1 producer on a topic and different number of consumers consuming that topic. I'm measuring the time between the creation of the message and its reception by the consumer.

The tests are done on a cluster of 3 nodes all connected to each others. Each broker is embedded in a JBoss. I used a cluster of 3 nodes because that's our current production setup. I'm challenging this setup because we have few consumers and producers (less than 50 each time) and we are using message grouping but I need to do POC on a setup with only 2 nodes in active/standby mode.

The producer is targeting always the same node and the consumers are connected to the other 2 nodes randomly.

We can see that for all cases, Artemis is slightly slower than ActiveMQ Classic. I'm wondering if this is something expected.

Answer 1

Generally speaking, ActiveMQ Artemis is notably faster than ActiveMQ "Classic" due to the significant architectural differences between them. In short, ActiveMQ Artemis was designed to be completely non-blocking and performs very well at scale compared to ActiveMQ "Classic".

However, in this case you are not testing the brokers at scale. You are testing one producer and a "different number" of consumers. This is certainly not the kind of production use-case that would warrant a cluster of 3 brokers. A single broker on modest or even minimal hardware would almost certainly suffice.

Even if you push the client count to around 50 I still think one live node would suffice. You definitely want to use just one live node if you're using message grouping. See the documentation for important details about clustered message grouping.

It's also important to keep in mind that you must compare "apples to apples" in terms of each broker's configuration. This is not necessarily trivial, especially when dealing with clusters. You did not share your broker configurations so I can't comment on whether or not they are functionally equivalent or at least as close to functionally equivalent as possible. There are many different configuration-specific reasons why one broker might perform better than the other in certain use-cases.

Over the last several years SoftwareMill has published benchmarks for popular message brokers for the persistent, replicated queue use-case. The last time both ActiveMQ "Classic" and Artemis were tested was in 2017. Here are the results. Since then SoftwareMill no longer tests ActiveMQ "Classic".

Answer 2

Performance testing is very complicated and if you do not control all aspects of the environment (network packet tuning, OS tuning, etc) it is very difficult to determine if one messaging product out performs another.

Also, echoing @Justin Bertram's comment that configuring two products for similar message flow characteristics requires expertise in both products, the message flow pattern, and the client app coding.

Lastly, when it comes to distributed systems and products-- often times you are hitting environment limits way before product limits-- things like disk perf, network perf, or app latency often limit out performance results. Esp for the 80% of enterprise workloads, which are not at massive scale.

Best advise-- understand your expected message flow pattern, and message size(s). Performance test that scenario to the point you know when you need to scale.