0

My web application needs to process a text file (size close to 200MB) for reading some relevant data. This text file needs to be processed monthly once, which means the contents of this text file varies monthly. I would like to know what is the best practice to implement the processing of this text file.

  1. Shall i allow the user to upload this 200mb file via the application itself and then process it?

  2. Configure the java webapplication to read the file from a predefined directory in the tomcat server.(Still i will present a UI to the user to initiate the processing, say e.g.'Start Processing')

I will be using JSaPAR library for processing my text file as it is a flat file with fixed positions.

user504342
  • 945
  • 2
  • 16
  • 36
active_coder
  • 79
  • 1
  • 2
  • 13
  • usually the 2nd option is the preferred one during such kinds of backend processing. the UI should be as light as possible with all the heavy processing done in the middletier of the app. – vikeng21 Jun 01 '14 at 10:44
  • @vikeng21 Thanks for the swift response. If I go with the 2nd option, then can I use a Servlet to accomplish this? Or is there any other library or methodology i should adhere to? How can I configure the predefined folder from where my Servlet should read the file? Any thoughts on this? – active_coder Jun 01 '14 at 10:47
  • you can implement it in many ways. one option is you can define a predefined path to a folder in a property file and later you can load that property file and access that in your code to fetch files from that path. other option may be to configure this path from the UI of your web app which you can store in the db and later do the accessing part. but at end of the day it all depends on the requirement of your customer which way they prefer right. – vikeng21 Jun 01 '14 at 10:56
  • Customer don't have anything specific, so I was thinking to propose this as as 'Admin' activity, where the Admin should place the text file in the predefined folder within the server machine. While accessing the respective module page in the UI, it can list whether there are any files to process. User can then click on the 'Start Processing' button or something to trigger the text file processing job. I think i can do the processing within the Struts Action itself by calling the respective APIs of the JaSPAR library. – active_coder Jun 01 '14 at 11:09
  • configuring path from the UI should be an Admin activity and i agree with you on this but placing the files in that path should not be a manual activity but it should be automated one. think on that a bit but otherwise you are on the right path. – vikeng21 Jun 01 '14 at 11:16
  • I don't think that I can automate the placement of files to the pre-defined folder because I am not sure from where customer is generating the text file. ie why initially I thought of Option 1, where customer always have an option to upload the file from the GUI itself, even if the file is residing in the computer from where he is accessing the app. But giving an option to upload a 200MB file is really a dumb idea.. So the only option is to force the customer to place the text file in a predefined folder from where the app can process it. – active_coder Jun 01 '14 at 11:28

1 Answers1

0

Let me answer the question myself.

A pointed out by vikeng21 I will also prefer the 2nd option. I implemented in such a way that, I have made the folder in which the files to be placed configurable via a property file. Also added options in the GUI which displays the files available for the system to process, by reading the pre-configured folder from the property file. If there are no files available in the folder, the GUI will also displays the absolute folder path to which the customer needs to place the files, so that the app can read and render the file info. A process link is also rendered against each file name, upon clicking on which the control transfers to an action class, which reads the file using java IO framework and then process the same using JSAPAR library.

active_coder
  • 79
  • 1
  • 2
  • 13