4

I'm building a Java application that will allow our users to load a list of files and have those files transferred to our server for video encoding. I've already built an API for managing the files before and after they've been transferred, but I need to decide on a good transfer protocol for actually moving the files.

Right now I'm leaning towards using the Apache Commons Net ( see: http://commons.apache.org/net/ ) package along with FTP to move the files from the client computer to the server. Once there I'll use secure API calls to move the files to wherever they need to go.

Is this the best route? Is there a better way to reliably transfer large (1 GB+) files? Is there a way to resume a broken download using this methodology? I'd like to avoid traditional HTTP POST requests as they're unreliable and cannot resume broken uploads.

Thanks!

FunnyLookinHat
  • 569
  • 1
  • 6
  • 13
  • I don't think you can resume broken connections with your approach. Do you think it supports resume? – phillip Feb 16 '11 at 18:28
  • I've had a lot of success using Commons Net for FTP, the only big hiccup I've had (and this may be because I'm using an older version) is that on the ls command if you're not in the right transfer mode (passive, non-passive) it just hangs... no exception, no crash, just... hangs... – Shaded Feb 16 '11 at 18:29
  • As far as I know you can't resume broken connections with Commons Net - so that's one reason I'm asking for suggestions... :) Also - thanks for the tip with passive vs. non-passive - probably saved me quite a few headaches. – FunnyLookinHat Feb 16 '11 at 18:30

2 Answers2

2

You didn't mention if using Amazon S3 is an option for your solution, but they do offer native partial upload support. The basic workflow is:

  1. Create an upload-placeholder and hold on to the response key
  2. Upload chunks -- can be concurrent and retried as necessary
  3. Use the response key to combine the chunks into a single file

Their SDK offers built-in file slicing and chunk upload.

Even if S3 is not the final location, you could use S3 as an upload holding pen and download the file at your convenience for permanent storage.

Uriah Carpenter
  • 6,656
  • 32
  • 28
  • Ah - well we do use Rackspace CloudFiles - which will support Chunked uploads soon (I just checked their roadmap). I hadn't even thought of that. Thanks for the idea! – FunnyLookinHat Feb 17 '11 at 14:02
0

Your understanding of HTTP post is not exactly correct. HTTP standard doesn't restrict range requests to only GET method - one can use them with POST or PUT as well.

Also if you have control over both client and server-side script, you can post both data chunk and the StartAt position as a separate parameter. On the server you check the parameter and append the received data at specified position.

Eugene Mayevski 'Callback
  • 45,135
  • 8
  • 71
  • 121
  • Ah yeah - I could definitely manually manage the data that way - but I was hoping to not have to do too much server-side (i.e. - hasn't the wheel been invented enough for me to just use someone else's wheel?) :) – FunnyLookinHat Feb 16 '11 at 20:17