Creating a Listening Service In Java

Question

I have a webapp with an architecture I'm not thrilled with. In particular, I have a servlet that handles a very large file upload (via commons-fileupload), then processes the file, passing it to a service/repository layer.

What has been suggested to me is that I simply have my servlet upload the file, and a service on the backend do the processing. I like the idea, but I have no idea to go about it. I do not know JMS.

Other details: - App is a GWT app split into the recommended client/server/shared subpackages, using an MVP architecture. - Currently, I am only running in GWT hosted mode, but am planning to move to Tomcat in the very near future.

I'm perfectly willing to learn whatever I need to in order to get this working (in fact, that's the point of writing the app). I'm not expecting anyone to write code for me, but can someone point me in the right direction to get started?

score 1 · Answer 1 · answered Aug 26 '13 at 07:27

This sounds interesting and familiar to me :). We do it in the similar way.

We have our four projects, all four projects includes file upload and file processing (Image/Video/PDF/Docs) etc. So we created a single project to handle all file processing, it is something like below:

All four projects and File processor use Amazon S3/Our File Storage for file storage so file storage is shared among all five projects.

We make request to File Processor providing details in XML via http request which include file-path on S3/Stoarge, aws authentication details, file conversion/processing parameters. File Processor does processing and puts processed files on S3/Storage, constructs XML with processed files details and sends XML via response.

We use Spring Frameowrk and Tomcat.

score 1 · Answer 2 · answered Jun 07 '11 at 17:33

1

There are many options for this scenario, but the simplest may be just copying the uploaded file to a known location on the file system, and have a background daemon monitor the location and process when it finds it.

answered Jun 07 '11 at 17:33

tylermac

499
2
8

Is this considered the "standard" way (i.e. if I got a job with a Fortune 500 company, is this the way they would do it)? If not, I might as well learn the "right" way. – Jason Jun 07 '11 at 17:40
In my opinion (and experience), many of them do :-) If my non-functional requirements were not too strict (say, regarding scalability, security, failover, transactions), I'd definitely choose this over JMS and get a solution that is simple to deploy and maintain. On the other hand, if my architecture would demand any such NFRs, I'd probably start looking for a sutiable JMS provider that'd do things for me. – david a. Aug 26 '13 at 08:35

score 1 · Answer 3 · answered Jun 07 '11 at 17:59

@Jason, there are many ways to solve your problem. i) Have dump you file data into Database with column type BLOB. and have a DB polling thread(after a particular time period) polls table for newly inserted file . ii) Have dump file into file system and have a file montioring process.

Benefit of i) over ii) is that DB is centralized and fast resource where as file systems are genrally slow and non-centalized in nature.

So basically servlet would dump either to DB or file system. Now about who will process that dumped file:- a) It could be either montioring process as discussed above or b) you can use JMS which is asynchronous in nature what it means servlet would put a trigger event in queue which will asynchronously trigger new processing thread. Well don't introduce JMS in your system unnecessarily if you are ok with monitoring process.

I'm kind of partial to taking the opportunity to learn JMS, actually. Threads worry me in a webapp scenario (probably unnecessarily if I were to be careful, but still). — Jason, Jun 07 '11 at 18:08
@jason: sure JMS can be good idea if you wish to. Just remember don't ever send file content as JMS payload to Queue instead only send file reference in queue....otherwise ur queue wud frequently blotted and running into many issues... — ag112, Jun 08 '11 at 18:03

score 0 · Answer 4 · edited May 23 '17 at 12:26

Since this is foremost a learning exercise, you need to pick an easy to use JMS provider. This discussion suggested FFMQ just one year ago.

Since you are starting with a simple processor, you can keep it simple and use a JMS Queue.

In the simplest form, each message send by the servlet has to correspond to a single job. You can either put the entire payload of the upload in the message, or just send a filename as reference to the content in the message. These are details you can refactor later.

On the processor side, if you are using Java EE, you can use a MessageBean. If you are not, then I would suggest a 3 JVM solution -- one each for Tomcat, the JMS server, and the message processor. This article includes the basics of a message consuming client.

Creating a Listening Service In Java

4 Answers4