0

My application needs to work as middleware where it has got orders(in form of xml) from various customers which contains the supplier id. Once it get the order, it needs to send order request to different suppliers in the form of xml.i am double minded about three aspects of it. Here they are:-

Questions:

  1. What i am planning at high level is as soon as request come, put it on jms queue.(Now i am not sure should i create queue for each supplier or one queue should be sufficient. I think one queue will be sufficient. as maintaining large number of queues will be overhead.). Advantage of maintaining separate queue per supplier is message can be processed faster as there will be separate producer on each queue.

  2. Before putting the object on queue i need to do some business validations. Also the structure of input xml i am receiving and output xml i need to send to supplier is different. For this i am planning to convert the input xml to java object then put on queue so that validation can be done with ease at consumer side. Another thought is dont convert the xml into java object, just get all elements value thru xpath/xstream api and validate them and put xml string as it is on queue because. Then at consumer side convert xml to java object then to different xml format. Is there a way of doing it?

  3. Now my requirement is consumer on queue process the messages on queue after every 5 hours and send the xml request to suppliers. I am planning to use quartz scheduler here which will pick the job one by one and send to corresponding supplier based on supplierId. Here is my question is if my job pick the message one by one and then send it to supplier. it will be too slow . I am planning to handle it where quartz job create ThreadPool with size of say ten threads at time which concurrently process the messages from queue(So here will be multiple consumers on queue. I think thats not valid for queue. Do i need topic here instead of queue?). Is second approach is better or there is some better than this?

i am expecting a load of 50k request per hour which mean around 15 request per second

M Sach
  • 33,416
  • 76
  • 221
  • 314
  • You haven't provided , what is the load . metrics – Mani Apr 03 '14 at 17:04
  • @Mani provided the load metric – M Sach Apr 03 '14 at 17:09
  • So in a peek time ( 4:59th hour) you would be excepting 250k objects in queue ? . Do you need to modify the format of XMl while sending back to Supplier. Reason of this in that case there is no point of storing as it in XML. And each supplier may except different format unless no contract between the middle tier to supplier – Mani Apr 03 '14 at 17:12
  • @Mani you are right i need to modify the format of xml while sending back to supplier.Mani i edited my second point based on your last comment.Good question. – M Sach Apr 03 '14 at 17:33

2 Answers2

1

Your basic requirement is ,

  1. Get order from customer as XML ( you have not told how you are receiving)
  2. Do basic Business validation .
  3. Send the Orders to Suppliers

And you would be excepting 50k Request ( You haven't provided the approximate an Order size). Assuming your Average order size 10K, it would be around 500 MB required just to hold it in Queue ( irrespective of number of queues) . i am not sure which environment you are running.

For Point #1 I would choose single Queue instead of multiple Queue - Choose the appropriate persistent store. I am assuming you would be using Distributed Queue , so that it can be easily scale while adding clusters.

For Point #2 I would be converting in POJO (Your own format ) and perform business validation. So that later if you want to extend the business validation to ruler or any other conversion it would be easy to maintain. - basically get the input in any form ( XML / POJO / JSON ...) and convert into Middle format ( you can write custome validator / conversion utility on top of Middle fomart) . And have Keep Mappings between the Common format to input as well output. So that you can write formatters and use them. which will not impact in future while changing format for any specific supplier. Try to externalize the format mapping.

For Point # 3 in your case, A Order needs to be processed by only once. So i would go with Queue. and you can have multiple Message Listeners . Message listeners deliver order in asynchronous. So you can have multiple Listeners for an Queue. And each listeners would run separate thread. Is there a problem to send the orders as soon as it received ? It would be good for you as well as the supplier to avoid heavy load at particular time.

Mani
  • 3,274
  • 2
  • 17
  • 27
  • can you join me for quick chat? – M Sach Apr 03 '14 at 18:12
  • let me try. Will give it a shot – M Sach Apr 03 '14 at 18:14
  • started chat. did you get my message on chat? – M Sach Apr 03 '14 at 18:22
  • are u there? i have joined chat room – M Sach Apr 03 '14 at 18:25
  • do you have skype id? – M Sach Apr 03 '14 at 18:27
  • A queue can have max producer . You said "queue can have multiple consumers" – M Sach Apr 03 '14 at 18:32
  • I tried to join . unfortunately my company proxy doesnt allow that url – Mani Apr 03 '14 at 18:34
  • you also said "Is there a problem to send the orders as soon as it received ? It would be good for you as well as the supplier to avoid heavy load at particular time.". If thats is the case we can avoid queue altogether. Is n't it? – M Sach Apr 03 '14 at 18:34
  • I think we can continue here only after couple of comments it will take us to chat automatically – M Sach Apr 03 '14 at 18:35
  • Yes and No - Assume you are avoiding queue. and sending as soon as receive. what if while sending to one of the suplier - your machine went down / jvm crashed. how you will retrive the order – Mani Apr 03 '14 at 18:36
  • In that case how jms queue will help? I mean will it not crash too? – M Sach Apr 03 '14 at 18:38
  • you can have transaction in queue. only you commit the transaction the object will remove else it will stay . so if jvm crashed the object will be redeliverd – Mani Apr 03 '14 at 18:39
  • Then when a transaction should be commited? – M Sach Apr 03 '14 at 18:40
  • You said "queue can have multiple consumers". But a queue can have max one consumer(producers can be multiple). Is n't it? – M Sach Apr 03 '14 at 18:41
  • When you complete the order. mean when you send the order successfully to suppliers. check your APP server docs how to handle transaction. In simple MessageListerner , it would be removing while go out of onMessage method. and you can ack the request – Mani Apr 03 '14 at 18:42
  • You said "queue can have multiple consumers". But a queue can have max one consumer(producers can be multiple). Is n't it?So how multiple producers will process the message from single queue? Thats why i was thinking of Topic instead of queue? – M Sach Apr 03 '14 at 18:44
  • i meant- multiple Message Listeners . you can have multiple Message Listeners in single JVM, As i mentioned any only one time the message will be delivered – Mani Apr 03 '14 at 18:46
  • see this like http://stackoverflow.com/questions/17336036/what-is-the-difference-between-a-messagelistener-and-a-consumer-in-jms . Message listeners supports asynchronous . as soon as it seems the message it would be delivered . and you can have n number of listeners. But you can have only one Consumer ( sycnronous). i will edit my answer to reflect correctly – Mani Apr 03 '14 at 18:51
  • you mean to say that i will register 10(any random number) listener on queue with single consumer where each listener will run its own thread. Is that right? – M Sach Apr 03 '14 at 19:10
  • you also said in first point " I am assuming you would be using Distributed Queue". How distributed queue will help here? – M Sach Apr 03 '14 at 19:11
  • yes , you can have n number of Listerners. And regarding DTQ. Assume you are running in clustered environment. Your queue needs to be clustered in all Managed servers - to do that you have to use DTQ. So that later if the load increase you can just add another manager server into cluster and boost the speed and manage the load – Mani Apr 03 '14 at 19:26
0

Since you are the middleware, you should handle data quick at the point of contact, to get your hands free for more incoming requests. Therefore you must find a way to distinquish the incoming data as quick and memory low as possible. Leave the processing of the data to modules more specific to the problem. A receptionist just directs the guests in the right spot.

If you really have to read and understand the received data in your specialized worker later on, use a threadpool. This way you can process the data parallelly without too much worry about outofmem. Just choose your pool size smartly and use only one. You could use a listener pattern to signal new incoming data to the worker multiton. You should avoid jaxb or better the complete deserialization of the data if possible. It eats up memory like hell.

I would not use jmx because you "messages" are relevant for only one listener.

If it is possible send the mail as soon as the worker is done with its work. If not, use a storage. This way you can later proove you processed the data and if something went wrong or you have to update your software, you do not have to worry about volatile data.

Hannes
  • 2,018
  • 25
  • 32