Uploading files through a play framework app to S3 without "touching" the disk

Question

I have to implement a server that lets some users upload possibly huge files (multi gigabytes worth of files) to S3 (they do not know it is S3 though). I have already implemented something similar in Python and a basic implementation using Play framework.

Right now my uploading function looks like this:

public Result uploadFile() {
      List<Http.MultipartFormData.FilePart> files = request().body().asMultipartFormData().getFiles();
      if (CollectionUtils.isNotEmpty(files)) {
          Bucket bucket = MY_BUCKET;
          UUID timeuuid;
          Promise<UploadResult> promise;
          ObjectNode responseMessage = Json.newObject();  
          for (Http.MultipartFormData.FilePart file : files) {
              timeuuid = TimeUUID.now();
              promise = Promise.promise(new UploadFunction(timeuuid, bucket, file.getFile(), obtainS3Connection()));
              responseMessage.set(file.getFilename(), TextNode.valueOf(timeuuid.toString()));
          }
          return accepted(responseMessage);
      } else {
          return badRequest("files empty");
      }
}

I am not worried about the upload code itself and actually I already have implemented it many times. What I am worried about is making it so that Play! does not save Files to disk and that the files sent only "pass through" my server without being written to disk (even as temp files). I have some processing involved and sometime "Client side" encryption of some content by I can do that as a Stream.

I was wondering if Play already does that? Or how would I achieve that?

What I am assuming Play (and Django in my previous version of this service) does is:

Upload --> Play       MyProcessing -------> S3
               \     /
                \   /
                 Disk

What I would like it to do :

Upload --> Play --> MyProcessing -------> S3

The reason I want to do this is that I want the potential to deploy my service on instances with very small disks and I also want the guarantee that nobody can find the unencrypted file (if I choose to encrypt it) once uploaded.

UPDATE: I am not giving up ... just yet.

I have found a few possible solution that unfortunately use Scala. They all revolve around the same theme as this discussion: Play 2.x : Reactive file upload with Iteratees

I was wondering if this would be possible using Java (I am the only one in my company that knows Scala so I cannot roll out something for which I would be the only person able to do maintenance on it ... because sometimes I wish to take breaks or go on other projects).

With files of that size and no disk, you will not have enough memory and you will need to open an incoming stream and an output stream. You will also need a reasonable buffer and some error handling in cases where the input arrives faster than the output is transmitted. So overall more complex. I would suggest looking for a way to create a file in S3 from code first. Clearly "UploadFunction" expects to deal with the underlying filesystem. — Daniel Langdon, Jul 17 '14 at 03:31

score 1 · Answer 1 · answered Sep 02 '14 at 10:54

I have never done this with Play however I have done something similar in Java. The route to success for us was to directly post files to S3. Basically what you do is you generate an upload form for the user where the post URL is not provided by your service but by Amazon and the users browser is directly uploading to S3.

On upload success S3 will then return an empty document with status code 200 when the object is successfully uploaded. In our case we perform the post call via jQuery and wait for the jQuery success handler on the post and only after that we update our systems database. Alternatively it is possible to specify a redirect form field with an URL where the user gets redirected to after successful upload.

Details on this can be found here: http://doc.s3.amazonaws.com/proposals/post.html

Uploading files through a play framework app to S3 without "touching" the disk

1 Answers1