0

I need a solution for the following scenario which is similar to a queue:

I want to write messages to a queue continuously. My message is very big, containing a lot of data so I do want to make as few requests as possible. So my queue will contain a lot of messages at some point. My Consumer will read from the queue every 1 hour. (not whenever a new message is written) and it will read all the messages from the queue.

The problem is that I need a way to read ALL the messages from the queue using only one call (I also want the consumer to make as few requests to the queue as possible).

A close solution would be ActiveMQ but the problem is that you can only read one message at a time and I need to read them all in one request.

So my question is.. Would there be other ways of doing this more efficiently? The actual thing that I need is to persist in some way messages created continuously by some application and then consume them (also delete them) by the same application all at once, every 1 hour.

The reason I thought a queue would be fit is because as the messages are consumed they are also deleted but I need to consume them all at once.

Georgiana_M
  • 47
  • 1
  • 7
  • How many messages are you consuming every hour? Do they need to be consumed in order? What kind of processing work does the client application do when it consumes a message? Does it require lots of resources (e.g. CPU, disk, memory, etc.)? Please elaborate. – Justin Bertram Dec 06 '18 at 17:04
  • I will give an example:The app is an online shop.Each product has a brand.An admin can change the price for all the products that have a certain brand.(so change the price for 10.000 products for example at once).All these product's info changed so they need to be updated on an external merchant,so I need to store them temporarly somewhere.Every hour I take those products and make a call to an external merchant with them.After each call they should be deleted.Right now, for each product that is to be updated is created and then removed an object in another table in DB which isn't efficient. – Georgiana_M Dec 06 '18 at 17:49
  • So another aproach would be using a queue but it is taking a lot of time to make 10.000 requests for writing each message and 10.000 for reading. – Georgiana_M Dec 06 '18 at 17:52
  • What exactly is involved with the "call to an external merchant"? Is that a HTTP request, a REST microservice, etc.? Do you have any concern that increasing the throughput of your application might overwhelm the "external merchant"? – Justin Bertram Dec 06 '18 at 18:25
  • Also, do you know for certain that you need a way to solve this problem more efficiently than with an ActiveMQ broker (whether that's the 5.x broker or the newer Artemis broker)? Have you tested this with ActiveMQ? If so, what were the results, and what performance numbers do you need? – Justin Bertram Dec 06 '18 at 18:28

1 Answers1

0

I think there's some important things to keep in mind as you're searching for a solution:

  1. In what way do you need to be "more efficient" (e.g. time, monetary cost, computing resources, etc.)?
  2. It's incredibly hard to prove that there are, in fact, no other "more efficient" ways to solve a particular problem, as that would require one to test all possible solutions. What you really need to know is, given your specific use-case, what solution is good enough. This, of course, requires knowing specifically what kind of performance numbers you need and the constraints on acquiring those numbers (e.g. time, monetary cost, computing resources, etc.).
  3. Modern message broker clients (e.g. those shipped with either ActiveMQ 5.x or ActiveMQ Artemis) don't make a network round-trip for every message they consume as that would be extremely inefficient. Rather, they fetch blocks of messages in configurable sizes (e.g. prefetchSize for ActiveMQ 5.x, and consumerWindowSize for ActiveMQ Artemis). Those messages are stored locally in a buffer of sorts and fed to the client application when the relevant API calls are made to receive a message.
  4. Making "as few requests as possible" is rarely a way to increase performance. Modern message brokers scale well with concurrent consumers. Consuming all the messages with a single consumer drastically limits the message throughput as compared to spinning up multiple threads which each have their own consumer. Rather than limiting the number of consumer requests you should almost certainly be maximizing them until you reach a point of diminishing returns.
Justin Bertram
  • 29,372
  • 4
  • 21
  • 43