0

We need to be able to supply big files to our users. The files can easily grow to 2 or 3GB. These files are not movies or similiar. They are software needed to control and develop robots in an educational capacity.

We have some conflict in our project group in how we should approach this challenge. First of all, Bittorrent is not a solution for us (despite the goodness it could bring us). The files will be availiable through HTTP (not FTP) and via a filestream so we can control who gets access to the files.

As a former pirate in the early days of the internet i have often struggled with corrupt files and using filehashes and filesets to minimize the amount of redownload required. I advocate a small application that downloads and verifies a fileset and extracts the big install file once it is completely downloaded and verified.

My colleagues don't think this is nessecary and point to the TCP/IP protocols inherit capabiltities to avoid corrupt downloads. They also mention that Microsoft has moved away from a downloadmanager for their MSDN files.

Are corrupt downloads still a widespread issue or will the amount of time we spend creating a solution to this problem be wasted, compared to the amount of people who will actually be affected by it?

If a download manager is the way to go, what approach would you suggest we take?

-edit- Just to clearify. Is downloading 3GB of data in one chunk, over HTTP a problem OR should we make our own EXE that downloads the big file in smaller chunks (and verifies them).

tshepang
  • 12,111
  • 21
  • 91
  • 136
mfriis
  • 217
  • 3
  • 17

1 Answers1

1

You do not need to go for your own download manager. You can use some really smart approach.

  1. Split files in smaller chunks, let's say 100MB each. So even if a download is corrupted, user will end-up downloading with that particular chunk.

  2. Most of web servers are capable of understanding and treating/serving range headers. You can recommend the users to use download manager / browser add-ons which can use this capacity. If your users are using unix/linux systems, wget is such a utility.

  3. Its true that TCP/IP has capacities of preventing corruption but it basically assumes that network is still up and accessible. #2 mentioned above can be one possible work-around to the problems where network was completely down in middle of download.

  4. And finally, it is always good to provide file hash to your users. This is not only to ensure the download but also to ensure the security of the software that you are distributing.

HTH

Community
  • 1
  • 1
Nehal Dattani
  • 933
  • 6
  • 15
  • We dont want the user to manually enqueue 20 files for download and manually check the filehash on each of those. The user is not tech savy. If this is needed we will need to do it with an application for them. – mfriis Jan 31 '13 at 13:33
  • consider 20 zip files and one click of DownThemAll! (https://addons.mozilla.org/en-US/firefox/addon/downthemall/) – Nehal Dattani Feb 01 '13 at 12:53
  • Our users/visitors come from all over the world, using all kinds of different OSes and browsers. A browser plugin is not really of interrest. Im looking for technical insight into either a premade exe that downloads our files OR someone saying it's no problem downloading 3GB in one chunk. – mfriis Feb 04 '13 at 11:25