0

I'm creating a proof of concept implementation in which I need to send files (sometimes large ones, up to 400Mb) embedded in messages using AMQP 1.0. I decided to use Apache's qpid proton client lib and qpid C++ broker.

The test setup has 3 parts: A sender program, the broker, and a receiver program. I'm using SSL for transport encryption and SASL for authentication. All parts run on the same box, connected through the loopback interface.

Everything works fine, except that the performance when sending files from the sender to the broker is really bad. Example: I sent a 140Mb message, and it took 90 seconds from the sender to the broker. The same message took only 2 seconds to deliver from the broker to the receiver.

The queue is not persistent, so no I/O bottlenecks.

I know that AMQP is not designed to send large messages, but seeing that it can actually deliver it fast enough on the receiving end, I figured it's capable of meeting my requirements, and I might be doing something wrong on the sending side.

So, the questions are:

  1. What could be causing this difference?
  2. Am I doing something wrong or is this the expected behavior?
  3. Is there any way to optimize the performance on the sending side?

I'm going crazy about this issue, so any help or idea that can point me to a solution is very welcomed. Thanks in advance!!!

Gabriel T
  • 23
  • 5

1 Answers1

1

AMQP can be used to send large message but how that gets done will depend a lot on the client and server implementations. If sending a large message it can be a good idea to lower the max frame size value to something smaller than the default which will usually be something like 2 GB frames (I don't know what the Qpid C++ client or broker do here) as that can allow the IO layer to send chunks of the delivery as opposed to waiting for the full thing to be written by the client application.

The client API can also be a factor as it needs a means of allowing for writing of partial bits of a larger message without waiting for the full value to be fed into the client sender (again I don't know if that's possible in the C++ client). The proton-c engine implementation combined with the API that I see in the docs would suggest to me that the client needs the full message loaded into memory and then inside proton will create another copy of the message for the framing layer and possible more copying depending on how framing ends up being done (one frame or many). I'd guess this has a lot to do with the overall sending time.

I'd advise against using the Qpid broker as it isn't actively maintained so you won't likely see any fixes or updates if you run into issues. An actively maintained broker such as ActiveMQ Artemis might be a better choice for a broker that can deal with large messages.

There is a Java client implementation at Qpid that does have an API purpose built for sending large messages which you can find here. There is an example of sending a file here.

Tim Bish
  • 17,475
  • 4
  • 32
  • 42
  • Thanks for the detailed answer! The sending time is measured from first to last partial (32Kb) block received by the broker, so I assume any memory copy inside the client lib should happen before that, and I think it's not part of the delay. I'll give your suggestions a try. – Gabriel T Apr 12 '23 at 07:51
  • I tried different values of max frame size (100Mb, 50Mb, 10Mb, 32Kb) and the result is the same on all cases, no impact in performance. – Gabriel T Apr 12 '23 at 09:50
  • 1
    Artemis broker seems to work way better, same 140Mb file took lees than 10 seconds. I'll consider this solved. Thanks again! – Gabriel T Apr 12 '23 at 13:41