12

I'm working with a setup involving many clients PCs and some server machines. I need to organize a reliable and fast method of file transfer between these PCs, that will be initiated by C# apps running on both. Any client may want to send/receive data from any server. The options are:

  • FTP - Use FtpWebRequest or SOSFTP to upload files onto an FTP server. The server checks for new files on its filesystem and does the required steps.

  • SCP - Secure file transfer. Usage same as FTP but increased security between machines. Probably slower than FTP.

  • DropBox / Box.Net - Use an online cloud storage solution with a library such as SharpBox. May be free / paid. May be less secure considering the said party has your files.

  • UDP - Use a library such as EME or GoAnywhere to transfer data from PC to PC directly via UDP. Probably faster but probably more unreliable since it uses custom technology.

What do you recommend?

James Z
  • 12,209
  • 10
  • 24
  • 44
Robin Rodricks
  • 110,798
  • 141
  • 398
  • 607

4 Answers4

10

I have a super-biased opinion, coming from a company that makes file transfer software, so I'm putting the disclaimer front and center to take everything I say with a grain of salt. ;-)

If you are sharing very large files to multiple endpoints, you are well served with one of two things:

  1. An MFT (Managed File Transfer) solution
  2. A product or API that uses a UDP-based protocol

1 - The MFT solutions typically use TCP-based transfer (like FTP) as the transport mechanism, but usually include additional reliability mechanisms not found in pure TCP. They will also include tools to schedule and organize transfers between many endpoints. To the best of my knowledge, there are not any significant MFT solutions that are available for free or low cost, but there are a plethora of enterprise-grade MFT solutions out there at the expected costs.

2 - For several years now, some companies have built proprietary control and reliability mechanisms into protocols built on top of UDP. By creating this custom layer, UDP (which has no reliability of its own) actually becomes MORE reliable and can have more features than TCP-based transfer can ever have, while not being adversely affected by packet loss and latency (read as: typically much faster transfer speeds). This protocol is not necessarily independent of MFT, either (it will inherently have some MFT characteristics and vendors might bake it into an MFT solution), but is potentially usable for its raw protocol (via API) and without all the UI trappings of a traditional "MFT solution".

There ARE open-source UDP-based file transfer protocols out there like Tsunami or UDT. They are not particularly well-maintained or feature-rich (there's some of my bias!) and as far as I know, none have a native C# library. On the same note, though, none of the commercial offerings that I'm aware of have a native C# library either, though wrappers are available.

If the kinds of transfers you describe are going to be a regular and ongoing part of the organization's daily activities, I have a hard time recommending anything beyond what I mention above. And to do it "right" (or at least without a non-trivial development project) you will probably want a commercial solution.

Greg Pettit
  • 10,749
  • 5
  • 53
  • 72
  • 4
    Why don't you recommend your commercial solution after so much advertising? – Robin Rodricks Jul 13 '12 at 10:03
  • 4
    There are too many MFT solutions to list. As for UDP-based file transfer, there are a few and the one I work for is in my profile for all to see! Would've been disingenuous to not put the disclaimer. However, I tried to provide information that doesn't favour one particular product, even those created by the company I work for. For obvious reasons file transfer is interesting to me so I wanted to offer my opinion and experience, that's all! – Greg Pettit Jul 13 '12 at 17:16
  • @GregPettit Would you recommend a MFT solution as well for many smaller files ( a. from 1 MegaByte to 5MB or b. from 15MB to 35MB). Each file is different than the other and has to be sent to a certain client somewhere in the world within less than a few minutes. Does the solution to pick change if you would have to sent several hundred files per minute. – surfmuggle Oct 04 '13 at 20:46
  • 2
    @threeFourOneSixOneThree - if you have *many* smaller files, a big savings will come from having them zipped before sending. If the MFT zips them before sending (autozip), there will still be a savings in time. If it can start transferring before the zip is even complete (progressive transfers), the situation is even better. If it has progressive transfers and acceleration (assuming the link is 10Mbps+; anything lower won't matter much) that is your best scenario. – Greg Pettit Oct 17 '13 at 05:49
4

I found it interesting that GoAnywhere is mentioned here. It is considered an MFT solution. I've used MFT for several years and found it very effective in doing exactly what you are describing here without having to write C# code. It's not free but I think it is well worth the cost.

It allows you to automate and schedule the various transfer protocols such as FTP, SFTP, FTPS, PGP, HTTPS, and also across network connections to multiple shared pc's and servers. MFT also allows me to work on formatting changes if I need to massage the data to work with different applications. Run queries to extract just the data I need from various data sources. It also logs all transactions so that I can verify that it has sent the files and can be setup to alert me with an email if there are any errors. I found it to be the perfect tool for all my data transfer requirements.

I am NOT an employee of any MFT software companies but a very happy customer that loves the product. It has made my work life SO much easier. There's a great example of how it helps IT staff here at http://blog.linomasoftware.com/2012/01/24/managed-file-transfer-changed-my-life/

Daniel Cheney
  • 41
  • 2
2

You could use the streaming feature of WCF

m0sa
  • 10,712
  • 4
  • 44
  • 91
2

As i understand what you have written, the data is shared to everyone client as server. So why not consider using torrent protocol. Here is a open source library. http://www.mono-project.com/MonoTorrent

Icy Creature
  • 1,875
  • 2
  • 28
  • 53
  • 1
    Torrents work best when there are multiple seeds and peers. For a single data source to a single client, there's little benefit over TCP-based transfer. But if Geotarget has a scenario in which multiple clients could peer to one another, it's a worthy idea. – Greg Pettit Jul 13 '12 at 03:43