efficiently pass files from webserver to file server

Question

i have multiple web server and one central file server inside my data center. and all my Web server store the user uploaded files into central internal file server.

i would like to know what is the best way to pass the file from web server to file server in this case?

as suggested i try to add more details to question:

the solution i came up was: after receiving files from user at web server, i should just do an Http Post to the file server. but i think there is some thing wrong with this because it causes large files to be entirely loaded into memory twice: (once at web server and once at file server)

Answer 1

Is your file server just another windows/linux server or is it a NAS device. I can suggest you number of approaches based on your requirement. The question is why d you want to use HTTP protocol when you have much better way to transfer files between servers.

HTTP protocol is best when you send text data as HTTP itself is based on text.From the client side to Server side HTTP is used as that is the only available option for you by our browsers .But between your servers ,I feel you should use SMB protocol(am assuming you are using windows as it is tagged for IIS) to move data.It will be orders of magnitude faster as much more efficient to transfer the same data over SMB vs HTTP.

And for SMB protocol,you do not have to write any code or complex scripts to do this.As provided by one of the answers above,you can just issue a simple copy command and it will happen for you.

So just summarizing the options for you (based on my preference)

1. Let the files get upload to some location on the each IIS web server e.g C:\temp\UploadedFiles . You can write a simple 2-3 line powershell script which will copy the files from this C:\temp\UploadedFiles to \FileServer\Files\UserID\\uploaded.file .This same powershell script can delete the file once it is moved to the other server successfully.

E.g script can be this simple and easy to make it as windows scheduled task

$Destination = "\\FileServer\Files\UserID\<FILEGUID>\"
New-Item -ItemType directory -Path $Destination -Force
Copy-Item -Path $Source\*.* -Destination $Destination -Force

This script can be modified to suit your needs to delete the files if it is done :)

2. In the Asp.net application ,you can directly save the file to network location.So in the SaveAs call,you can give the network path itself. This you have to make sure this network share is accessible for the IIS worker process and also has write permission.Also in my understanding asp.net gets the file saved to temporary location first (you do not have control on this if you are using the asp.net HttpPostedFileBase or FormCollection ). More details here

You can even run this in an async so that your requests will not be blocked

if (FileUpload1.HasFile) 
          // Call to save the file.
          FileUpload1.SaveAs("\\networkshare\filename");

https://msdn.microsoft.com/en-us/library/system.web.ui.webcontrols.fileupload.saveas(v=vs.110).aspx

3.Save the file the current way to local directory and then use HTTP POST. This is worst design possible as you are first going to read the contents and then transfer it as chunked to other server where you have to setup another webservice which recieves the file.The you have to read the file from request stream and again save it to your location. Am not sure if you need to do this.

let me know if you need more details on any of the listed method.

Answer 2

Or you just write it to a folder on the webservers, and create a scheduled task that moves the files to the file server every x minutes (e.g. via robocopy). This also makes sure your webservers are not reliant on your file server.

Answer 3

Assuming that you have an HttpPostedFileBase then the best way is just to call the .SaveAs() method.

You need the UNC path to the file server and that is it. The simplest version would look something like this:

public void SaveFile(HttpPostedFileBase inputFile) {
  var saveDirectory = @"\\fileshare\application\directory";
  var savePath = Path.Combine(saveDirectory, inputFile.FileName);
  inputFile.SaveAs(savePath);
}

However, this is simplistic in the extreme. Take a look at the OWASP Guidance on Unrestricted File Uploads. File uploads can be the source of many vulnerabilities in your application.

You also need to make sure that the web application has access to the file share. Take a look at this answer

Creating a file on network location in asp.net

for more info. Generally the best solution is to run the application pool with a special identity which is only used to access the folder.

Answer 4

the solution i came up was: after receiving files from user at web server, i should just do an Http Post to the file server. but i think there is some thing wrong with this because it causes large files to be entirely loaded into memory twice: (once at web server and once at file server)

I would suggest not posting the file at once - it's then full in memory, which is not needed.

You could post the file in chunks, by using ajax. When a chunk receives at your server, just add it to the file.

With the File Reader API, you could read the file in chunks in Javascript.

Something like this:

/** upload file in chunks */
function upload(file) {

    var chunkSize = 8000; 

    var start = 0;

    while (start < file.size) {
        var chunk = file.slice(start, start + chunkSize);
        var xhr = new XMLHttpRequest();
        xhr.onload = function () {
            //check if all chunks are and then send filename or send in in the first/last request.
        };
        xhr.open("POST", "/FileUpload", true);
        xhr.send(chunk);

        start = end;

    }
}

Answer 5

It can be implemented in different ways. If you are storing files in files server as files in file system. And all of your servers inside the same virtual network

Then will be better to create shared folder on your file server and once you received files at web server, just save this file in this shared folder directly on file server.

Here the instructions how to create shared folders: https://technet.microsoft.com/en-us/library/cc770880(v=ws.11).aspx

Answer 6

Just map a drive

I take it you have a means of saving the uploaded file on the web server's local filesystem. The question pertains to moving the file from the web server (which is probably one of many load-balanced nodes) to a central file system all web servers can access it.

The solution to this is remarkably simple.

Let's say you are currently saving the files some folder, say c:\uploadedfiles. The path to uploadedfiles is stored in your web.config.

Take the following steps:

1. Sign on as the service account under which your web site executes

2. Map a persistent network drive to the desired location, e.g. from command line:

   NET USE f: \\MyFileServer\MyFileShare /user:SomeUserName password
   

3. Modify your web.config and change c:\uploadedfiles to f:\

Ta da, all done.

Just make sure the drive mapping is persistent, and make sure you use a user with adequate permissions, and voila.