0

AMQ 5.7 here. I have inherited a very old set of Java applications that use an ActiveMQ (AMQ) broker to communicate between each other as well as with clients outside this ecosystem. Hence the AMQ broker has several hundred (!!!) queues on it, and there are literally hundreds of JMS clients living on dozens of different servers (each) publishing messages to and consuming messages from these queues. Its a rats nest.

I have a queue, let's call it shouldBeDead, that should not be receiving any more messages. It was deprecated at some point in time and nothing should be sending messages to that queue anymore. However periodically it gets bombarded with hundreds of messages from something/somewhere. It has no consumers on it (which is correct; my suite of Java apps no longer use it anywhere in their code and so nothing is listening on that queue to consume messages off of it).

To complicate matters even more, this is an old AMQ version suffering from this known UI bug where the TLDR; is: I need to upgrade the AMQ instance to > 5.12.x. However for reasons outside the context of this question, I can't upgrade AMQ at this time. So whereas I was hoping to browse the enqueued messages on shouldBeDead and drill down into them to get information about them, I can't even view them in the AMQ web UI nor in the app logs.

I'm simply trying to figure out where these messages are coming from!

Doing a network analysis might be fruitful but is outside the scope of my skill sets and these messages flood the queue at seemingly random times without any discernable pattern.

I'm hoping there's some AMQ command-line fu I can do to inspect queue metadata, perhaps peek into KahaDB or any other type of magic I can do to look at these messages and/or get tracking/client-side info out of them.

Worst case I know I can deploy some code changes to add a shouldBeDead listener/consumer back in and to log the messages, however I'm really trying to do this without making any code changes. Any ideas/thoughts? Thanks in advance!

hotmeatballsoup
  • 385
  • 6
  • 58
  • 136

1 Answers1

1

I would approach the problem via the JMX console.

Here is a list of diagnostic informations available via JMX, and here is a way how to access this data via command line. By knowing the exact details of your problem, like:

  • how predictable these bursts are
  • roughly how many messages are being
  • sent how many connections you have

you need to capture the relevant info to catch your producer.

Without knowing the details of course, one idea that might work:
Run a bash script periodically that checks the size of the queue (like Destination/EnqueueCount) + periodically saves all the Active connections. When you detect the spike in the messages on the deprecated queue, look back what connection just appeared at that time (Connection/RemoteAddress).

Gergely Bacso
  • 14,243
  • 2
  • 44
  • 64
  • Thanks so much @Gergely (+1)! Just to play Devil's advocate here, if I *were* to try and go with code changes (that is: change one of my Java apps to listen on the `shouldBeDead` queue and consume messages off of them) is it possible (using JMS/AMQ Java APIs) to get `Connection/RemoteAddress` info out of the messages it consumes? Cursory Googling shows `JMSServerControl controller = (JMSServerControl) JMX.newMBeanProxy(server,objectName,JMSServerControl.class,true )` might hold some promise? Thanks again! – hotmeatballsoup Oct 18 '18 at 14:02
  • 1
    JMSServerControl is a HornetMQ class. Don't get confused with the AMQ Artemis project either, that is also HornetMQ based, I do not think you are running that one. Regarding the programmatic solution in general: I don't have any clear idea yet. Maybe if you share more info. My questions above, + "can you lose those messages? Can you print those messages?" etc whatever else you might think is helpful. – Gergely Bacso Oct 18 '18 at 14:26
  • Thanks again @Gergely (+1) -- good catch on JMSServerControl being a HornetMQ class! I definitely like a JMX-based approach, I think I'm looking for the `Connection` bean (which will help me establish client connection info and where messages are coming from), but is there any way (via JMX) to map a connected client to the queues/topics it producing to? Ideally there would be a JMX bean or maybe a `Connection` bean property that shows which clients are sending messages to which queues...is this possible? Thanks again! – hotmeatballsoup Oct 18 '18 at 14:46
  • Unfortunately I do not think so. I looked for it yesterday and I could not find a direct connection. That is why I suggested looking for bursting clients. There _might be_ some magic JMX parameter giving you a straight answer directly, but I am not aware of it. Have not used AMQ in the past 4 years, so memories are fading. – Gergely Bacso Oct 18 '18 at 14:51