2

We are migrating from IBM MQ to Amazon MQ, at least we would like to do so. The problem is Amazon MQ is having bad performance when using JMS producer to put a large message on a queue compared to IBM MQ.

All messages are persistent and the system is High Available regarding IBM MQ, and Amazon MQ is multi AZ.

If we put this size of XML files to IBM MQ (2 cpu and 8GB RAM HA instance) we have this performance:

256 KB = 15ms
4,6 MB = 125ms
9,3 MB = 141ms 
18,7 MB = 218ms
37,4 MB = 628ms
74,8 MB = 1463ms

If we put the same files on Amazon MQ (mq.m5.2xlarge = 8 CPU and 32 GB RAM) or ActiveMQ we have this performance:

256 KB = 967ms
4,6 MB = 1024ms
9,3 MB = 1828ms 
18,7 MB = 3550ms
37,4 MB = 8900ms
74,8 MB = 14405ms

What we also see is that IBM MQ has equal response times for sending a message to a queue and getting a message from a queue, while Amazon MQ is real fast in getting a message (e.g. just takes 1 ms), but very slow on sending.

On Amazon MQ we use the OpenWire protocol. We use this config in Terraform style:

resource "aws_mq_broker" "default" {
  broker_name                = "bernardamazonmqtest"
  deployment_mode            = "ACTIVE_STANDBY_MULTI_AZ"
  engine_type                = "ActiveMQ
  engine_version             =  "5.15.10"
  host_instance_type         =  "mq.m5.2xlarge"
  auto_minor_version_upgrade =  "false"
  apply_immediately          =  "false"
  publicly_accessible        =  "false"
  security_groups            = [aws_security_group.pittensbSG-allow-mq-external.id]
  subnet_ids                 = [aws_subnet.pittensbSN-public-1.id, aws_subnet.pittensbSN-public-3.id]

  logs {
    general = "true"
    audit   = "true"
  }

We use Java 8 with JMS ActiveMQ library via POM (Maven):

<dependency>
    <groupId>org.apache.activemq</groupId>
    <artifactId>activemq-client</artifactId>
    <version>5.15.8</version>
</dependency>
<dependency>
    <groupId>org.apache.activemq</groupId>
    <artifactId>activemq-pool</artifactId>
    <version>5.15.8</version>
</dependency>

In JMS we have this Java code:

private ActiveMQConnectionFactory mqConnectionFactory;
private PooledConnectionFactory mqPooledConnectionFactory;
private Connection connection;
private Session session;
private MessageProducer producer;
private TextMessage textMessage;
private Queue queue;

this.mqConnectionFactory = new ActiveMQConnectionFactory();
this.mqPooledConnectionFactory = new PooledConnectionFactory();
this.mqPooledConnectionFactory.setConnectionFactory(this.mqConnectionFactory);

this.mqConnectionFactory.setBrokerURL("ssl://tag-1.mq.eu-west-1.amazonaws.com:61617");

this.mqPooledConnectionFactory.setMaxConnections(10);

this.connection = mqPooledConnectionFactory.createConnection());
this.connection.start();

this.session = this.connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
this.session.createQueue("ExampleQueue");

this.producer = this.session.createProducer(this.queue);
long startTimeSchrijf = 0;
startTimeWrite= System.currentTimeMillis();
producer.send("XMLFile.xml");  // here we send the files
logger.debug("EXPORTTIJD_PUT - Put to queue takes: " + (System.currentTimeMillis() - startTimeWrite));

// close session, producer and connection after 10 cycles

We also have run the performance test as a Single Instance AmazonMQ. But same results. We have also run the performance test with a mq.m5.4xlarge (16 cpu, 96 GB RAM) engine but still no improvement of the bad performance.

Performance test configuration: We first push the messages(XML files) according above one by one to a queue. We do that 5 times. After 5 times we read those messages(XML files) from the queue. We call this 1 cycle.

We run 10 cycles one after another, so in total we have pushed 300 files to the queue and we have getted 300 files from the queue.

We run 3 tests in parallel: One from AWS Region Londen, one from AWS Region Frankfurt in a different VPC and 1 from Frankfurt in the same VPC as the Amazon MQ broker and in the same subnet. Alle clients run on an EC2 instance: m4.xlarge.

If we run a test with only one VPC for example only the local VPC which is in the same subnet as the AmazonMQ broker the performance improves and we have these results:

256 KB = 72ms
4,6 MB = 381ms
9,3 MB = 980ms 
18,7 MB = 2117ms
37,4 MB = 3985ms
74,8 MB = 7781ms

The client and server are in the same subnet, so we have nothing to do with firewalling etc.

Maybe somebody can tell me what is wrong, and why we have such a terrible performance with Amazon MQ or ActiveMQ?

extra info: Response times are measured in the JMS Java app with Java starttime just before the producer.send('XML') and just endtime just after the producer.send('XML'). Difference is the recorded time. Times are average times over 300 calls.
IBM MQ server is located in our datacenter, and client app is running at a server in the same datacenter.

extra info test: The jms app starts create connectionFactory queues sessions. Then it uploads the files to MQ 1 by 1. This is a cycle, then it run this cycle 10 times in a for lus without opening or closing sessions queues or connectionfactorys. Then all 60 messages are read from queue and written to files on the local drive. Then it closes the connection factory and session and producer/consumer. This is one batch. Then we run 5 batches. So between the batches connectionFactory, queue, session are recreated.

In response to Sam: When I also execute the test with the same size of files like you did Sam I approach the same response times, I set the persistence mode also to false value between () :

500 KB = 30ms (6ms)
1 MB = 50ms (13ms)
2 MB = 100ms (24ms)

I removed the connection pooling and I set concurrentStoreAndDispatchQueues="false" The system I have used broker: mq.m5.2xlarge and client: m4.xlarge.

But if I test with bigger files, this are the response times:

256 KB = 72ms
4,6 MB = 381ms
9,3 MB = 980ms 
18,7 MB = 2117ms
37,4 MB = 3985ms
74,8 MB = 7781ms

I am having a very simple requirement. I have a system what puts messages on a queue and the messages are get from the queue by another system, sometimes at the same time sometimes not, sometimes there are 20 or 30 messages on the system before they get unloaded. Thats why I need a queue and messages must be persistent and it must be a Java JMS implementation.

I think Amazon MQ might be a solution for small files but for big files it is not. I think we have to use IBM MQ for this case which has better performance. But one important thing: I tested IBM MQ only on premis in our LAN. We tried to test IBM MQ on Amazon but we didn't succeed yet.

Ben
  • 594
  • 1
  • 9
  • 24
  • That's a very good question, could you please elaborate on your way you benchmark it (What is measured? Where the start and end of time measuring? Are these average times for one message or is it for whole batch?), and where's IBM MQ instance located? Is it in your local network? Also, in you test, do you open a connection on each file sent or you push/pull 300 files within the same connection? Sorry for a lot of refinements I ask for, but it would be helpful if you update your question with these details. – Anton Goncharov Mar 31 '20 at 10:20
  • Thanks Anton for response,I have updated my request to answer your questions – Ben Apr 01 '20 at 10:27

2 Answers2

1

I tried to reproduce the scenario you were testing. When I ran a JMS client in the same VPC as the AmazonMQ broker for mq.m5.4xlarge broker with an Active and Standby instance, I see the following roundtrip latencies - measuring the moment from which a producer sends a message to the moment when consumer receives the message.

2MB - 50ms
1MB - 31ms
500KB - 15ms

My code just created a connection and a session. I did not use a PooledConnectionFactory (stating this as a matter of fact, not saying/suspecting that's the cause). Also it is better to strip down the code to bare minimum in order to establish a baseline and remove noise when doing performance testing. That way, when you introduce additional code, you can easily see if the new code introduced a performance issue. I used the default broker configuration.

In ActiveMQ, there is a concept of Fast Producer and Fast Consumer, this means, if consumer can process the messages at the same rate as the Producer, the broker transfers the message from producer to consumer via memory and then it writes the message to disk. This is the default behavior and is controlled by a broker configuration setting named concurrentStoreAndDispatch which is true (default)

If consumer is unable to keep up with producer, and thus becomes a "slow" consumer and with the concurrentStoreAndDispatch flag set to true, you take a performance hit.

ActiveMQ provides advisory topics which you can subscribe to detect slow consumers. If in fact, you detected that the consumer is slower than the producer, it is better to set concurrentStoreAndDispatch flag to false to get better performance.

Sam Andaluri
  • 36
  • 2
  • Hi Sam, thanks very much for your response. See above in my question where I respond to your answer. – Ben Apr 19 '20 at 18:26
  • Ben, what is the minimum latency expectation for these messages? One alternative would be to upload large messages to S3 and place the S3 pointer on the Amazon MQ queue. When the consumer picks up the message, takes the S3 pointer and gets the message from S3. – Sam Andaluri Apr 20 '20 at 19:58
  • Hi Sam thx, we have hired an external company and together with AWS engineers they advice us about next steps, and they will evaluate my test. – Ben Apr 22 '20 at 16:38
0

I don't get any response. I think its because there is no solution for this performance problem. Amazon MQ is a cloud service and mabye thats the reason why performance is this bad. IBM MQ is a different architecture, and it is on premise.

I have to investigate the performance of ActiveMQ some more before I can tell what exactly the reason is for this problem.

Ben
  • 594
  • 1
  • 9
  • 24