Stream processing

Question

Have a requirement that when user is uploading a file it should work in a following manner

1)File upload dialog (in browser) is presented to the user. User picks a file. 2) Application should load only first x number of records (e.g. lets say total # of records are 100 then get first 10) and user will have a chance to do visual review of records (read only view). 3)User then decides one of two things : "Click on Submit" which will take in all the data and streams to the server, Or if s/he click on "Next" s/he can review next 10 records etc.

Is Scalaz-stream a good fit as a over all solution and in particular for doing 2) and 3) from above? To get only partial data and pause the stream then continue, consume, and repeating the process?

Answer 1

No, scalaz-stream is not a good idea. The Play! framework has its own framework with the Enumerator, Enumeratee, and Iteratee classes which can be used for asynchronous processing of streams, and the file upload code is already built to use it.

You have two options:

One, use HTML5 and front-end Javascript to get access to the file. This will only work in the newest browsers. This is the only option if you don't want any of the file uploaded until the user chooses "Submit".

Two, incrementally parse the upload as it comes to the server using the Enumerator framework, and respond over long-polling AJAX/Comet/Websocket to the front-end Javascript with a subset of records as they are parsed. The Iteratee that is parsing the incoming upload will have to pause and wait for further input from the front-end. This solution would be complex and would suffer from issues with the browser timing out.

Neither of these are a very good idea. It would be much simpler to have the entire file upload all at once, have the parsed records fed back to the front-end afterwards, and have the "Submit" button actually function as a "Save" button to tell the server to keep the received upload. Unless you are shoving 100 MiB+ Excel files up a mobile connection, this is likely the easiest and most compatible solution.