0

Problem:

Process a backlog of messages where each message has three headers "service", "client", and "stream". I want to process the backlog of messages with maximum concurrency, but I have some requirements:

  • Only 10 messages with the same service can be processing at once.
  • Only 4 messages with the same service AND client can be processing at once.
  • All messages with the same service AND client AND stream must be kept in order.

Additional Information:

I've been playing around with "maxConcurrentConsumers" along with the "JMSXGroupID" in a ServiceMix (Camel + ActiveMQ) context, and I seem to be able to get 2 out of 3 of my requirements satisfied.

For example, if I do some content-based routing to split the backlog up into separate "service" queues (one queue for each service), then I can set the JMSXGroupID to (service + client + stream), and set maxConcurrentConsumers=10 on routes consuming from each queue. This solves the first and last requirements, but I may have too many messages for the same client processing at the same time.

Please note that if a solution requires a separate queue and route for every single combination of service+client, that would become unmanageable because there could be 10s of thousands of combinations.

Any feedback is greatly appreciated! If my question is unclear, please feel free to suggest how I can improve it.

рüффп
  • 5,172
  • 34
  • 67
  • 113
David Gordon
  • 21
  • 1

1 Answers1

0

To my knowledge, this would be very hard to achieve if you have 10k+ combos.

You can get around one queue per service/client combo by using consumers and selectors. That would, however, be almost equally hard to deal with (you simply don't create 10k+ selector consumers unharmed and without significant performance considerations), if you cannot predict in some way a limited set of service/client active at once.

Can you elaborate on the second requirement? Do you need it to make sure there are some sense of fairness among your clients? Please elaborate and I'll update if I can think of anything else.

Update: Instead of consuming by just listening to messages, you could possibly do a browse on the queue, looping through the messages and pick one that "has free slots". You can probably figure out if the limit has been reached by some shared variable that keeps track given you run in a single instance.

Petter Nordlander
  • 22,053
  • 5
  • 50
  • 84
  • Thanks very much for your insight, @Petter! I am not very familiar with the selector concept, so that will be a great thing for me to research. – David Gordon Jan 06 '13 at 20:35
  • As for the second requirement, it has to do with SLA. The web services represented by the "service" header only allow a specified number of concurrent logins for each of the clients. – David Gordon Jan 06 '13 at 20:38
  • Selectors can be useful, as it's "like" SQL towards queues - but mind the performance impact. Adding some stuff in the answer. Not sure if it helps – Petter Nordlander Jan 06 '13 at 21:04
  • I think that selectors and/or the BrowsableEndpoint interface could really help. If I am able to achieve the implementation goal that way, I will be sure to post. – David Gordon Jan 06 '13 at 21:56
  • Sure, do so. It would be interesting :) – Petter Nordlander Jan 06 '13 at 22:01