0

I am using ActiveMQ to store messages to be used later. It is working as expected, but there is a specific scenario I need to fit which I cannot figure out. The short question is this.

Is there a way to run a query on the queue to find out all messages with certain header values?

The problem in detail is this :

So there is a set of data that is coming in multiple messages and the requirement is to use that data only after all messages for that has come in. So if the dataset has lets say 50 messages i need to wait for those 50 messages and then read them in. I am adding headers to each message to denote they belong to a certain set. Like "TotalSets"=50 , "SetId"=39 . I would like to write a thread that keeps tracing if all sets for a particular batch has arrived.

Justin Bertram
  • 29,372
  • 4
  • 21
  • 43
Som Bhattacharyya
  • 3,972
  • 35
  • 54
  • It sounds to me like you need a database rather than a message broker. See http://sensatic.net/messaging/messaging-anti-patterns-part-1.html for further discussion of this anti-pattern. – Justin Bertram Dec 18 '18 at 20:11
  • Well i get what you mean but doesn't apply in my case. First , i want a place to temporarily cache the mesages so other backend services can pick those up and save them to the database. So there isnt too much sense in it for me to have yet another database. – Som Bhattacharyya Dec 19 '18 at 06:42
  • And yet the fact remains that you can't query a message broker like you can a database which appears to be your goal. I don't see any problem with using one database as a temporary cache for another database, but if you don't want to do that for whatever reason then I would encourage you to not think about a message broker like you think about a database. They are designed differently to solve different problems. – Justin Bertram Dec 19 '18 at 15:08
  • Bingo ! Well the reason why i want to avoid the database is to avoid blocking. Today it is the blocking which is casuing our services to freeze. The reason i am thinking of the messaging platform is to have a place which can act as a very short term holding place for the messages and i will have a few other apps consuming messages from the queues per their capabilities. I understand you are trying to point me to a better place. Do you think maybe writing to a temporary nosql DB for cache will be a good idea ? – Som Bhattacharyya Dec 20 '18 at 07:10
  • What exactly is blocking in the database? Inserting the data? Sending durable messages will block as well so I'm not sure you're getting much benefit from that angle. You can send messages asynchronously which won't block, but then you won't get any guarantee that the data is persisted on the broker (a guarantee you *would* get with a database). It's nearly impossible to make general recommendations in this kind of forum because there are so many unknown details. All I can really say at this point is that you shouldn't treat a message broker like a database. – Justin Bertram Dec 20 '18 at 14:49
  • I can really use some discussion to solve the problem we have. But here is a summary. I have a client which today pumps out .net datasets(in the form of bytes) to our backend web services(written using the legacy asmx technology). The webservices then call another set of services that save the bytes to files. Then a service reads those files and save to db. Now with thousands of clients the web services are getting hit like never before. The consumption rate of web services is unable to match up with the sending rate of clients. So i was thinking of queues for temporary cache. – Som Bhattacharyya Dec 21 '18 at 06:32
  • 1
    You've got lots of layers here (client->web-service->file-service->database-service->database). There's just too many variables and unknown details to say what the real problem is and how to solve it. Adding a message broker to this picture may solve the problem or it just may mask it for a bit and eventually make it worse since it will add even more complexity to your code and environment. – Justin Bertram Dec 21 '18 at 16:27
  • i get it. Thanks. I will try to do a POC and weigh it out. Thanks. – Som Bhattacharyya Dec 28 '18 at 17:51

1 Answers1

1

NMS is a .NET equivalent to the JMS messaging API so the means of filtering messages is the same as in JMS, your subscription applies a JMS Message Selector when created to tell the broker what messages it is interested in. The session methods to create MessageConsumer instances have variants that accept the selector using the JMS defined syntax which is your means of filtering messages.

Tim Bish
  • 17,475
  • 4
  • 32
  • 42