How Can I Reduce The Memory Useage For a Huge File Transfer?

Question

I have to transfer some huge files (2GB-ish) to a web service:

public bool UploadContent(System.Web.HttpContext context)
{
     var file = context.Request.Files[0];
     var fileName = file.FileName;

     byte[] fileBytes = new Byte[file.ContentLength];
     file.InputStream.Read(fileBytes, 0, fileBytes.Length);
     client.createResource(fileBytes);
}

The HttpContext already has the contents of the file in File[0], but I can't see a way to pass those bytes to the createResource(byte[] contents) method of the web service without making a copy as a byte array... so I am eating memory like candy.

Is there a more efficient way to do this?

EDIT client.createResource() is part of a COTS product and modification is outside our control.

score 0 · Accepted Answer · answered Apr 27 '15 at 21:12

Rather than sending the whole bytes you can send the chunks of the files. Seek the file for step by step upload and merge the next chunk to already save bytes on server. You need to update your client.CreateResource method only if you're allowed to modify that method :)

Add following parameters: string fileName // To locate the file name when you start sending the chunks byte[] buffer // chunk that would be sent to server via webservice long offset // Information that will tell you how much data is already uploaded, so that you can seek the file and merge the buffer.

Now your method will look like:

public bool CreateResource(string FileName, byte[] buffer, long Offset)
{
    bool retVal = false;
    try
    {
        string FilePath = "d:\\temp\\uploadTest.extension";
        if (Offset == 0)
            File.Create(FilePath).Close();
        // open a file stream and write the buffer. 
        // Don't open with FileMode.Append because the transfer may wish to 
        // start a different point
        using (FileStream fs = new FileStream(FilePath, FileMode.Open,
            FileAccess.ReadWrite, FileShare.Read))
        {
            fs.Seek(Offset, SeekOrigin.Begin);
            fs.Write(buffer, 0, buffer.Length);
        }
        retVal = true;
    }
    catch (Exception ex)
    {
        // Log exception or send error message to someone who cares
    }
    return retVal;
}

Now to read the file in chunks from the InputStream of HttpPostedFile try below code:

public bool UploadContent(System.Web.HttpContext context)
{
    //the file that we want to upload
    var file = context.Request.Files[0];
    var fs = file.InputStream;

    int Offset = 0; // starting offset.

    //define the chunk size
    int ChunkSize = 65536; // 64 * 1024 kb

    //define the buffer array according to the chunksize.
    byte[] Buffer = new byte[ChunkSize];
    //opening the file for read.

    try
    {
        long FileSize = file.ContentLength; // File size of file being uploaded.
        // reading the file.
        fs.Position = Offset;
        int BytesRead = 0;
        while (Offset != FileSize) // continue uploading the file chunks until offset = file size.
        {
            BytesRead = fs.Read(Buffer, 0, ChunkSize); // read the next chunk 

            if (BytesRead != Buffer.Length)
            {
                ChunkSize = BytesRead;
                byte[] TrimmedBuffer = new byte[BytesRead];
                Array.Copy(Buffer, TrimmedBuffer, BytesRead);
                Buffer = TrimmedBuffer; // the trimmed buffer should become the new 'buffer'
            }
            // send this chunk to the server. it is sent as a byte[] parameter, 
            // but the client and server have been configured to encode byte[] using MTOM. 
            bool ChunkAppened = client.createResource(file.FileName, Buffer, Offset);
            if (!ChunkAppened)
            {
                break;
            }

            // Offset is only updated AFTER a successful send of the bytes. 
            Offset += BytesRead; // save the offset position for resume
        }
    }
    catch (Exception ex)
    {
    }
    finally
    {
        fs.Close();
    }
}

Disclaimer: I haven't tested this code. This is a sample code to show how large file upload can be achieved without hampering the memory.

Ref: Source article.

Sadly, `client.createResource()` is outside our purview. – Scott Baker Apr 27 '15 at 21:58 — Scott Baker, Apr 27 '15 at 21:58
Ugh! you have to live with it now. – vendettamit Apr 28 '15 at 12:59 — vendettamit, Apr 28 '15 at 12:59

How Can I Reduce The Memory Useage For a Huge File Transfer?

1 Answers1